Today, I'm moving to San Francisco. Moving to the United States to work on a startup has been one of my biggest dreams ever, and it became reality when my O-1 was approved shortly after my 25th birthday two months ago. I've been working on this for more than a decade, so today, I'd like to share my whole story up to this point.

I was born in Freiburg, a city in southern Germany. My parents are doctors and have always worked hard. Because of their jobs, we moved frequently, I grew up in Zurich, Switzerland and Innsbruck, Austria. I've had an interest in building since I was a kid. At twelve years old, I started extending my favorite video game to play with friends. Over time, I moved on to mobile and web application development, using any free minute I could get to learn more and work hard to get better. Little did I know that my fun hobby would turn into a career.

I've always been drawn to startups. Autonomy and ownership are two of the most important things I value. I went to high school close to Frankfurt and got my first summer internship as a software engineer in 2017, when I was 15 years old. I loved the experience so much that I swore to myself to learn harder than ever before to get a real job. A year later, I landed my first job as a software engineer at Hygraph. I worked harder than ever before, and joined the company full-time after graduating high school in 2019, watching the company grow from five people to over 70 in the span of a couple years.

During Covid, I moved to Munich where I've lived for the past five years. I studied at TUM and LMU, the two most prestigious universities in Germany, graduating from TUM in 2023. I tried building a startup with my best friend and even though it didn't work out at the time, I learned valuable lessons about the business environment and challenges software startups are still facing in Germany.

I took everything I learned up to that point and joined Inngest as a distributed systems engineer in early 2024. Through Inngest, I was able to visit San Francisco three times in the past two years and I fell in love with the city. Tech runs deep in San Francisco (you really can't escape it) and the energy and level of optimism in everyone I've met so far are unlike anywhere else. People work incredibly hard and builders are taken seriously, with lots of trust and support. In December 2024, I knew that I wanted to move to the US and I started the visa process that would take more than 18 months.

If I had told my fifteen-year-old self that within ten years, I would be living and working for a startup in Silicon Valley, I simply wouldn't have believed it. This is the power of pursuing a dream, joined with the opportunities presented by the US. I learned the value of working hard and never giving up, no matter how many roadblocks you find in your way. I also learned the value of focusing on what matters most and understanding opportunity cost. Time and attention are the scarcest resources in life.

Moving to the US is just the beginning. I want to work hard, learn from the best, grow, and help shape the future with products that have a real impact on people's lives. I deeply believe that now is the best time to build, in all of human history. The tools and resources we have at our disposal have never been more powerful. The question is no longer how to build, it's what to build, and why. The challenge is to assemble a world-class team and build the best possible product out there to solve real problems.

I'm incredibly excited for the next chapter and I'll keep sharing more on my journey.

I want to thank everyone without whom I wouldn't be where I am today.

First of all, I want to thank my amazing girlfriend for being supportive of this plan. I truly couldn't have done this without you. I want to thank my good friends Tim, Mohamad, Kai, Nico, Hendrik, Timo, Benjamin, Noelia, Paul, Lorenz, Miguel, and Philipp. You've always encouraged me to challenge myself and dream bigger.

From Inngest, I want to thank the entire team, with special thanks to Tony, Darwin, Albert, Jacob, Jakob, Lakshmi, Riadh, and Muzammil. Thank you for your trust and belief in me.

From Hygraph, I want to thank Michael, Daniel, Jonas, Fabi, Dino, Pablo, Jean, Julian, Frederik, and Larisa. Thank you for teaching me more in a couple years than I ever believed was possible.

From TUM, I want to thank Prof. Pramod Bhatotia for your support during my thesis and the research projects you invited me to help on.

I also want to thank Jennifer Li, Martin Casado, and Yoko Li from a16z, Ivan from Daytona, Andrea from Mendral, Jake from Railway, and Mathias from Netlify for supporting me with the visa.

Every second counts

This year, I got to work on numerous challenging and exciting projects, including rewriting nearly all of the internal systems ingesting billions of events and powering hundreds of millions of workflow runs every day. We've raised a $21m Series A earlier this year and are facing the best (tons of happy customers with amazing use cases) and worst (scaling challenges) effects of product market fit.

This August, I traveled to London to speak at GopherCon UK about the lessons I learned migrating mission-critical systems with zero downtime. This was my first talk in years, so I was reasonably nervous leading up to it. Judging from the feedback I got by attendees walking up to me after the session, I did better than I anticipated and I'm looking forward to return for another round next year!

In October, we held our annual team offsite in Boston. It was inspiring to see how much the team has grown this year (we more than doubled in size), and I can't wait to see everyone in person again soon. I also got to travel to San Francisco on numerous occasions and I always return more energized than I left.

As we're entering the last days of 2025, I'm recharging and looking ahead on to 2026. I'm confident that this will be the best year yet.

My primary focus for the architecture of this blog always has been the content backbone. Post content has been stored as Markdown files in a Git repository for the past 8 years and I do not expect this to change in the next 8 years either. Images and media are also stored in the repository using Git LFS, and hosted on my static subdomain with proper Cache-Control headers.

Previously, I spent a lot of effort on designing a polished landing page, including animations "inspired" by the best (are you really a web developer until you've attempted to copy Stripe?). For this iteration, my focus was on the content. As long as I found a way to get the Markdown files properly displayed, I'd be happy.

My good friend and frequent collaborator Tim switched his blog to Ghost, an open content and newsletter platform you can self-host. I trust his taste in software and share his idea about ownership, so I didn't think long before making this decision.

Self-hosting Ghost isn't a miracle, and their documentation contains everything you need. Where it gets a little more interesting is my way of syncing content: Most people probably just opt to author their posts in Ghost's fantastic editor (as did I, for the post you're reading at this very moment), yet I wanted the guarantee of having all my content on disk, neatly versioned and all.

Syncing from Markdown to Ghost

To get my 230+ existing posts onto Ghost in the first place, I had to write a script to convert the Markdown source to HTML and upload everything via the built-in Admin API. This was easier than I thought, and with the help of Claude Code, I got to the following solution:

#!/usr/bin/env python3
"""
Ghost CMS Sync Script
Syncs Markdown blog posts from content/posts/ to Ghost CMS.
"""

import os
import sys
import time
import glob as file_glob
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional, Tuple

import jwt
import requests
import frontmatter
import markdown2
from dotenv import load_dotenv


# Debug mode - set to True to see HTML output
DEBUG = os.getenv('DEBUG', 'false').lower() == 'true'


# Configuration
GHOST_API_URL = "https://YOUR_URL_HERE"
CONTENT_DIR = Path(__file__).parent.parent / "content" / "posts"
ENV_FILE = Path(__file__).parent.parent / ".env"

# External post types that should not be synced to Ghost
EXTERNAL_POST_TYPES = ['EXTERNAL']


class GhostAPIClient:
    """Client for interacting with Ghost Admin API."""

    def __init__(self, api_url: str, admin_api_key: str):
        self.api_url = api_url.rstrip('/')
        self.admin_api_key = admin_api_key

        # Parse the admin API key
        try:
            self.key_id, self.key_secret = admin_api_key.split(':')
        except ValueError:
            raise ValueError("Invalid GHOST_ADMIN_API_KEY format. Expected 'ID:SECRET'")

    def _generate_jwt_token(self) -> str:
        """Generate a JWT token for Ghost Admin API authentication."""
        iat = int(datetime.now(timezone.utc).timestamp())

        header = {
            'alg': 'HS256',
            'typ': 'JWT',
            'kid': self.key_id
        }

        payload = {
            'iat': iat,
            'exp': iat + 300,  # Token expires in 5 minutes
            'aud': '/admin/'
        }

        token = jwt.encode(payload, bytes.fromhex(self.key_secret), algorithm='HS256', headers=header)
        return token

    def _make_request(self, method: str, endpoint: str, data: Optional[Dict] = None) -> requests.Response:
        """Make an authenticated request to Ghost API."""
        token = self._generate_jwt_token()
        url = f"{self.api_url}{endpoint}"

        headers = {
            'Authorization': f'Ghost {token}',
            'Content-Type': 'application/json'
        }

        response = requests.request(method, url, json=data, headers=headers)
        return response

    def get_post_by_slug(self, slug: str) -> Optional[Dict]:
        """Get a post by its slug."""
        try:
            response = self._make_request('GET', f'/ghost/api/admin/posts/slug/{slug}/')
            if response.status_code == 200:
                return response.json()['posts'][0]
            elif response.status_code == 404:
                return None
            else:
                print(f"  Warning: Unexpected status {response.status_code} when fetching slug {slug}")
                return None
        except Exception as e:
            print(f"  Error fetching post by slug {slug}: {e}")
            return None

    def get_posts_by_title(self, title: str) -> List[Dict]:
        """Get all posts with a specific title."""
        try:
            # Use filter parameter to search by title
            response = self._make_request('GET', f'/ghost/api/admin/posts/?filter=title:\'{title}\'&limit=all')
            if response.status_code == 200:
                return response.json()['posts']
            else:
                print(f"  Warning: Unexpected status {response.status_code} when searching for title")
                return []
        except Exception as e:
            print(f"  Error searching posts by title: {e}")
            return []

    def delete_post(self, post_id: str) -> Tuple[bool, Optional[str]]:
        """Delete a post from Ghost."""
        try:
            response = self._make_request('DELETE', f'/ghost/api/admin/posts/{post_id}/')
            if response.status_code == 204:
                return True, None
            else:
                error_msg = response.json().get('errors', [{}])[0].get('message', 'Unknown error')
                return False, f"Status {response.status_code}: {error_msg}"
        except Exception as e:
            return False, str(e)

    def create_post(self, post_data: Dict) -> Tuple[bool, Optional[str]]:
        """Create a new post in Ghost."""
        try:
            response = self._make_request('POST', '/ghost/api/admin/posts/?source=html', {'posts': [post_data]})
            if response.status_code in [200, 201]:
                return True, None
            else:
                error_msg = response.json().get('errors', [{}])[0].get('message', 'Unknown error')
                return False, f"Status {response.status_code}: {error_msg}"
        except Exception as e:
            return False, str(e)

    def update_post(self, post_id: str, post_data: Dict) -> Tuple[bool, Optional[str]]:
        """Update an existing post in Ghost."""
        try:
            response = self._make_request('PUT', f'/ghost/api/admin/posts/{post_id}/?source=html', {'posts': [post_data]})
            if response.status_code == 200:
                return True, None
            else:
                error_msg = response.json().get('errors', [{}])[0].get('message', 'Unknown error')
                return False, f"Status {response.status_code}: {error_msg}"
        except Exception as e:
            return False, str(e)


def parse_markdown_file(file_path: Path) -> Optional[Tuple[Dict, str]]:
    """Parse a markdown file and return frontmatter and content."""
    try:
        with open(file_path, 'r', encoding='utf-8') as f:
            post = frontmatter.load(f)

        # Validate required fields
        required_fields = ['title', 'path', 'date', 'published']
        missing_fields = [field for field in required_fields if field not in post.metadata]

        if missing_fields:
            print(f"  ⚠ Skipping {file_path.name}: Missing fields {missing_fields}")
            return None

        return post.metadata, post.content
    except Exception as e:
        print(f"  ⚠ Error parsing {file_path.name}: {e}")
        return None


def convert_markdown_to_html(markdown_content: str) -> str:
    """Convert Markdown content to HTML, preserving code block language classes."""
    import re
    import html as html_lib

    # Extract fenced code blocks with language info
    code_blocks = []

    def extract_fenced_block(match):
        language = match.group(1) or ''
        code = match.group(2)
        # Use HTML comment as placeholder (won't be processed by markdown2)
        placeholder = f'<!--CODE_BLOCK_{len(code_blocks)}-->'
        code_blocks.append((language, code))
        return placeholder

    # Extract fenced code blocks (with or without language)
    fenced_pattern = r'```(\w+)?\n(.*?)```'
    temp_md = re.sub(fenced_pattern, extract_fenced_block, markdown_content, flags=re.DOTALL)

    # Convert markdown to HTML (without fenced-code-blocks extra since we handle it manually)
    # Note: 'code-friendly' removed to enable underscore-based emphasis (_italic_)
    extras = ['tables', 'break-on-newline']
    html = markdown2.markdown(temp_md, extras=extras)

    # Replace placeholders with properly formatted code blocks
    for i, (language, code) in enumerate(code_blocks):
        placeholder = f'<!--CODE_BLOCK_{i}-->'
        code_escaped = html_lib.escape(code.rstrip())

        if language:
            replacement = f'<pre><code class="language-{language}">{code_escaped}</code></pre>'
        else:
            replacement = f'<pre><code>{code_escaped}</code></pre>'

        html = html.replace(placeholder, replacement)

    return html


def extract_slug_from_path(path: str) -> str:
    """Extract slug from the path field."""
    # Handle full URLs (external posts)
    if path.startswith('http://') or path.startswith('https://'):
        # Extract the last segment from the URL path
        # e.g., 'https://www.inngest.com/blog/sharding-at-inngest' -> 'sharding-at-inngest'
        from urllib.parse import urlparse
        parsed = urlparse(path)
        slug = parsed.path.rstrip('/').split('/')[-1]
        return slug if slug else path

    # Handle local paths like '/blog/2024-12-30-post-title'
    # The path is like '/blog/2024-12-30-looking-back-on-2024'
    # We want to extract just '2024-12-30-looking-back-on-2024'
    # Ghost will add the /blog/ prefix via its routing configuration
    if path.startswith('/blog/'):
        return path[6:]  # Remove '/blog/' prefix

    # Fallback for other formats
    return path.lstrip('/')


def map_to_ghost_post(frontmatter: Dict, html_content: str) -> Dict:
    """Map frontmatter and content to Ghost post format."""
    slug = extract_slug_from_path(frontmatter['path'])

    # Map published boolean to status
    status = 'published' if frontmatter.get('published', True) else 'draft'

    # Get topics as tags (or empty list if not present)
    tags = frontmatter.get('topics', [])

    # Ensure tags is a list
    if not isinstance(tags, list):
        tags = [tags]

    # Generate keywords meta tag for code injection
    keywords = frontmatter.get('keywords', [])
    keywords_tag = None
    if keywords and isinstance(keywords, list) and len(keywords) > 0:
        # Filter out None values and ensure all keywords are strings
        keywords = [str(k) for k in keywords if k is not None]
        if keywords:  # Only create meta tag if there are valid keywords
            # Escape special characters for HTML attributes
            import html
            keywords_str = ', '.join(keywords)
            keywords_escaped = html.escape(keywords_str, quote=True)
            keywords_tag = f'<meta name="keywords" content="{keywords_escaped}" />'

    # Generate og:image meta tag for code injection
    ogimage_tag = None
    if 'ogImageUrl' in frontmatter and frontmatter['ogImageUrl']:
        import html
        ogimage_url = html.escape(str(frontmatter['ogImageUrl']), quote=True)
        ogimage_tag = f'<meta property="og:image" content="{ogimage_url}" />'

    ghost_post = {
        'title': frontmatter['title'],
        'slug': slug,
        'html': html_content,
        'status': status,
        'published_at': frontmatter['date'],
        'tags': tags
    }

    # Add feature image if present
    if 'headerImageUrl' in frontmatter:
        ghost_post['feature_image'] = frontmatter['headerImageUrl']
        ghost_post['feature_image_alt'] = frontmatter['title']

    # Add code injection if keywords or og:image exist
    codeinjection_tags = []
    if keywords_tag:
        codeinjection_tags.append(keywords_tag)
    if ogimage_tag:
        codeinjection_tags.append(ogimage_tag)

    if codeinjection_tags:
        ghost_post['codeinjection_head'] = '\n'.join(codeinjection_tags)

    return ghost_post


def sync_post_to_ghost(client: GhostAPIClient, file_path: Path, dry_run: bool = False, cleanup_duplicates: bool = False) -> str:
    """Sync a single post to Ghost. Returns status: 'created', 'updated', 'skipped', or 'error'."""
    # Parse the markdown file
    result = parse_markdown_file(file_path)
    if result is None:
        return 'error'

    frontmatter_data, markdown_content = result

    # Skip external post types
    post_type = frontmatter_data.get('type', '')
    if post_type in EXTERNAL_POST_TYPES:
        print(f"  ⊘ Skipping external post type: {post_type}")
        return 'skipped'

    # Convert markdown to HTML
    html_content = convert_markdown_to_html(markdown_content)

    # Map to Ghost post format
    ghost_post = map_to_ghost_post(frontmatter_data, html_content)
    slug = ghost_post['slug']
    title = ghost_post['title']

    # Debug output
    if DEBUG:
        print(f"\n  Slug: {slug}")
        print(f"  Title: {ghost_post['title']}")
        print(f"  Status: {ghost_post['status']}")
        print(f"  Tags: {ghost_post['tags']}")
        print(f"  HTML preview (first 500 chars):\n{html_content[:500]}\n")
        if '<img' in html_content:
            print(f"  ✓ Images found in HTML")
            # Extract and show image tags
            import re
            img_tags = re.findall(r'<img[^>]+>', html_content)
            for img in img_tags[:3]:  # Show first 3 images
                print(f"    {img}")
        else:
            print(f"  ✗ No images found in HTML")

    if dry_run:
        print(f"  [DRY RUN] Would sync: {slug}")
        # Check for duplicates in dry-run mode too
        posts_with_title = client.get_posts_by_title(title)
        if len(posts_with_title) > 1:
            print(f"  ⚠ Found {len(posts_with_title)} posts with title '{title}':")
            for p in posts_with_title:
                print(f"    - {p['slug']} (ID: {p['id']}, updated: {p['updated_at'][:10]})")
        return 'skipped'

    # Check if post exists by slug
    existing_post = client.get_post_by_slug(slug)

    # Also check for duplicates by title
    posts_with_title = client.get_posts_by_title(title)

    # Handle duplicates
    if cleanup_duplicates and len(posts_with_title) > 1:
        print(f"  ⚠ Found {len(posts_with_title)} duplicate posts with title '{title}'")
        # Sort by updated_at to keep the most recent
        posts_sorted = sorted(posts_with_title, key=lambda p: p['updated_at'], reverse=True)
        post_to_keep = posts_sorted[0]
        posts_to_delete = posts_sorted[1:]

        print(f"    Keeping: {post_to_keep['slug']} (ID: {post_to_keep['id']}, updated: {post_to_keep['updated_at'][:10]})")

        for post in posts_to_delete:
            print(f"    Deleting: {post['slug']} (ID: {post['id']}, updated: {post['updated_at'][:10]})")
            success, error = client.delete_post(post['id'])
            if success:
                print(f"    ✓ Deleted duplicate")
            else:
                print(f"    ✗ Error deleting duplicate: {error}")

        # Use the kept post as the existing post
        existing_post = post_to_keep

    if existing_post:
        # Update existing post
        ghost_post['updated_at'] = existing_post['updated_at']
        success, error = client.update_post(existing_post['id'], ghost_post)
        if success:
            return 'updated'
        else:
            print(f"  ✗ Error updating {slug}: {error}")
            return 'error'
    else:
        # Create new post
        success, error = client.create_post(ghost_post)
        if success:
            return 'created'
        else:
            print(f"  ✗ Error creating {slug}: {error}")
            return 'error'


def main():
    """Main function to sync all posts."""
    import argparse

    parser = argparse.ArgumentParser(description='Sync Markdown posts to Ghost CMS')
    parser.add_argument('--dry-run', action='store_true', help='Preview changes without syncing')
    parser.add_argument('--limit', type=int, help='Limit number of posts to process')
    parser.add_argument('--post', type=str, help='Sync only a specific post (filename)')
    parser.add_argument('--cleanup-duplicates', action='store_true', help='Detect and remove duplicate posts with the same title')
    args = parser.parse_args()

    # Load environment variables
    load_dotenv(ENV_FILE)

    api_key = os.getenv('GHOST_ADMIN_API_KEY')
    if not api_key:
        print("Error: GHOST_ADMIN_API_KEY not found in .env file")
        sys.exit(1)

    # Initialize Ghost API client
    try:
        client = GhostAPIClient(GHOST_API_URL, api_key)
    except ValueError as e:
        print(f"Error: {e}")
        sys.exit(1)

    # Find all markdown files
    if args.post:
        # Sync specific post
        post_path = CONTENT_DIR / args.post
        if not post_path.exists():
            print(f"Error: Post not found: {post_path}")
            sys.exit(1)
        markdown_files = [post_path]
    else:
        # Sort in reverse order (newest first)
        markdown_files = sorted(CONTENT_DIR.glob('*.md'), reverse=True)
        if args.limit:
            markdown_files = markdown_files[:args.limit]

    total_posts = len(markdown_files)

    cleanup_msg = " (with duplicate cleanup)" if args.cleanup_duplicates else ""
    if args.dry_run:
        print(f"[DRY RUN MODE] Processing {total_posts} posts from {CONTENT_DIR}{cleanup_msg}...\n")
    else:
        print(f"Processing {total_posts} posts from {CONTENT_DIR}{cleanup_msg}...\n")

    # Track statistics
    stats = {
        'created': 0,
        'updated': 0,
        'error': 0,
        'skipped': 0
    }

    # Process each file
    for i, file_path in enumerate(markdown_files, 1):
        print(f"[{i}/{total_posts}] {file_path.name}")

        status = sync_post_to_ghost(client, file_path, dry_run=args.dry_run, cleanup_duplicates=args.cleanup_duplicates)
        stats[status] += 1

        if status == 'created':
            print(f"  ✓ Created")
        elif status == 'updated':
            print(f"  ✓ Updated")
        elif status == 'skipped' and not args.dry_run:
            print(f"  - Skipped")

        # Small delay to avoid rate limiting (skip in dry-run)
        if not args.dry_run:
            time.sleep(0.1)

    # Print summary
    print("\n" + "=" * 50)
    print("Sync Summary:")
    print(f"  Created: {stats['created']}")
    print(f"  Updated: {stats['updated']}")
    print(f"  Skipped: {stats['skipped']}")
    print(f"  Failed: {stats['error']}")
    print(f"  Total: {total_posts}")
    print("=" * 50)


if __name__ == '__main__':
    main()

As you can see, this script

• fetches all Markdown files in a specific subdirectory

• converts the source to HTML

• uses the Admin API to create the post if not exists, or update existing posts

• cleans up duplicate posts

• provided keywords, tags, header images, social images, etc. in the expected format

And while this is cool, do you know what's even cooler? Automating it, of course! So I set up a GitHub Actions workflow to run the script every time content changed:

name: Sync Posts to Ghost CMS

on:
  push:
    branches:
      - main
    paths:
      - 'content/posts/**'
      - 'scripts/sync-to-ghost.py'
      - 'scripts/requirements.txt'
  workflow_dispatch:
    inputs:
      dry_run:
        description: 'Run in dry-run mode'
        required: false
        default: 'false'
        type: choice
        options:
          - 'true'
          - 'false'

jobs:
  sync-to-ghost:
    runs-on: ubuntu-latest
    permissions:
      contents: read

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: pip install -r scripts/requirements.txt

      - name: Sync posts to Ghost
        env:
          GHOST_ADMIN_API_KEY: ${{ secrets.GHOST_ADMIN_API_KEY }}
          PYTHONUNBUFFERED: 1
        run: |
          if [ "${{ github.event.inputs.dry_run }}" = "true" ]; then
            python scripts/sync-to-ghost.py --dry-run
          else
            python scripts/sync-to-ghost.py
          fi

      - name: Upload sync summary
        if: always()
        run: |
          echo "Sync completed. Check logs for details."

This is great for getting all content uploaded to Ghost, but what about using Ghost to author new posts while still relying on Markdown as the single source of truth? Enter the reverse sync process!

Syncing from Ghost to Markdown

Fetching and uploading all Markdown content to Ghost is nice, but it would be even better if I didn't have to enter a code editor to author my blog posts. Don't get me wrong, I love NeoVim now, but writing a blog is not the same as writing code to me, so I would strongly prefer a lighter experience.

After uploading all my initial posts I quickly thought: Why can't I run the same process in reverse? If I can turn Markdown into HTML, the opposite shouldn't be rocket science either. So I set out to create a reverse syncing script.

#!/usr/bin/env python3
"""
Ghost CMS Reverse Sync Script
Syncs posts from Ghost CMS to local Markdown files.
"""

import os
import re
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional
from urllib.parse import urlparse

import jwt
import requests
import frontmatter
from markdownify import MarkdownConverter
from bs4 import BeautifulSoup
from dotenv import load_dotenv


# Configuration
GHOST_API_URL = "https://YOUR_DOMAIN_HERE"
CONTENT_DIR = Path(__file__).parent.parent / "content" / "posts"
STATIC_DIR = Path(__file__).parent.parent / "content" / "static" / "posts"
ENV_FILE = Path(__file__).parent.parent / ".env"


class GhostAPIClient:
    """Client for interacting with Ghost Admin API."""

    def __init__(self, api_url: str, admin_api_key: str):
        self.api_url = api_url.rstrip('/')
        self.admin_api_key = admin_api_key

        # Parse the admin API key
        try:
            self.key_id, self.key_secret = admin_api_key.split(':')
        except ValueError:
            raise ValueError("Invalid GHOST_ADMIN_API_KEY format. Expected 'ID:SECRET'")

    def _generate_jwt_token(self) -> str:
        """Generate a JWT token for Ghost Admin API authentication."""
        iat = int(datetime.now(timezone.utc).timestamp())

        header = {
            'alg': 'HS256',
            'typ': 'JWT',
            'kid': self.key_id
        }

        payload = {
            'iat': iat,
            'exp': iat + 300,  # Token expires in 5 minutes
            'aud': '/admin/'
        }

        token = jwt.encode(payload, bytes.fromhex(self.key_secret), algorithm='HS256', headers=header)
        return token

    def get_all_posts(self) -> List[Dict]:
        """Fetch all posts from Ghost with pagination."""
        all_posts = []
        page = 1

        while True:
            token = self._generate_jwt_token()
            url = f"{self.api_url}/ghost/api/admin/posts/"
            params = {
                'limit': 50,
                'page': page,
                'formats': 'html',
                'include': 'tags,codeinjection_head'
            }
            headers = {
                'Authorization': f'Ghost {token}',
                'Content-Type': 'application/json'
            }

            try:
                response = requests.get(url, params=params, headers=headers)
                response.raise_for_status()
            except Exception as e:
                raise Exception(f"Failed to fetch posts (page {page}): {e}")

            data = response.json()
            posts = data.get('posts', [])
            all_posts.extend(posts)

            # Check if there are more pages
            meta = data.get('meta', {}).get('pagination', {})
            if page >= meta.get('pages', 1):
                break

            page += 1
            time.sleep(0.1)  # Small delay between requests

        return all_posts


class CustomMarkdownConverter(MarkdownConverter):
    """Custom converter that preserves code block languages."""

    def convert_pre(self, el, text, **kwargs):
        """Override pre tag conversion to preserve language classes."""
        if not text:
            return ''

        # Check if this pre contains a code element
        code = el.find('code')
        if code is not None:
            # Extract language from class attribute
            classes = code.get('class', [])
            if isinstance(classes, str):
                classes = classes.split()

            language = ''
            for cls in classes:
                if cls.startswith('language-'):
                    language = cls.replace('language-', '')
                    break

            # Get the code content
            code_text = code.get_text()

            # Return fenced code block with language
            # Ensure code_text ends with newline for proper fence formatting
            if not code_text.endswith('\n'):
                code_text += '\n'

            if language:
                return f'\n```{language}\n{code_text}```\n'
            else:
                return f'\n```\n{code_text}```\n'

        return super().convert_pre(el, text, **kwargs)


def convert_html_to_markdown(html: str) -> str:
    """Convert Ghost HTML to Markdown, preserving code block languages."""
    return CustomMarkdownConverter(
        heading_style="ATX",
        bullets="-",
        escape_asterisks=False,
        escape_underscores=False,
        strip=['script', 'style']
    ).convert(html)


def download_images(html: str, slug: str, dry_run: bool = False) -> None:
    """Download images from HTML to static directory."""
    soup = BeautifulSoup(html, 'html.parser')

    images_to_download = []
    for img in soup.find_all('img'):
        src = img.get('src')
        if not src or not src.startswith('https://static.brunoscheufler.com/posts/'):
            continue

        # Extract the full relative path from the URL
        # e.g., "https://static.brunoscheufler.com/posts/slug/gcloud/1.png"
        # -> extract "slug/gcloud/1.png"
        parsed = urlparse(src)
        url_path = parsed.path  # e.g., "/posts/provisioning-k8s-cluster/gcloud/1.png"

        # Remove '/posts/' prefix to get relative path
        if url_path.startswith('/posts/'):
            relative_path = url_path[7:]  # Remove '/posts/' -> "slug/gcloud/1.png"
        else:
            continue

        # Build target path preserving subdirectories
        target_path = STATIC_DIR / relative_path

        # Check if file exists
        if target_path.exists():
            print(f"    Image already exists: {relative_path}")
            continue

        images_to_download.append((src, target_path, relative_path))

    if not images_to_download:
        return

    if dry_run:
        # In dry-run mode, just log what would be downloaded
        for src, target_path, relative_path in images_to_download:
            print(f"    [DRY RUN] Would download: {relative_path}")
        return

    # Create parent directories and download
    for src, target_path, relative_path in images_to_download:
        try:
            target_path.parent.mkdir(parents=True, exist_ok=True)  # Create all parent dirs
            response = requests.get(src, timeout=10)
            response.raise_for_status()
            target_path.write_bytes(response.content)
            print(f"    Downloaded: {relative_path}")
        except Exception as e:
            print(f"    Warning: Failed to download {src}: {e}")


def extract_keywords_from_code_injection(codeinjection_head: Optional[str]) -> List[str]:
    """Extract keywords from Ghost code injection HTML."""
    if not codeinjection_head:
        return []

    soup = BeautifulSoup(codeinjection_head, 'html.parser')
    keywords_tag = soup.find('meta', {'name': 'keywords'})

    if keywords_tag:
        content = keywords_tag.get('content', '')
        if content:
            return [k.strip() for k in content.split(',')]

    return []


def extract_ogimage_from_code_injection(codeinjection_head: Optional[str]) -> Optional[str]:
    """Extract og:image URL from Ghost code injection HTML."""
    if not codeinjection_head:
        return None

    soup = BeautifulSoup(codeinjection_head, 'html.parser')
    ogimage_tag = soup.find('meta', {'property': 'og:image'})

    if ogimage_tag:
        content = ogimage_tag.get('content', '')
        if content:
            return content.strip()

    return None


def map_ghost_post_to_frontmatter(post: Dict) -> Dict:
    """Map Ghost post fields to frontmatter."""
    # Extract date from published_at or created_at
    published_at = post.get('published_at') or post.get('created_at')

    # Map status
    is_published = post.get('status') == 'published'

    # Extract tag names
    tags = post.get('tags', [])
    topics = [tag['name'] for tag in tags if isinstance(tag, dict)]

    # Build path with /blog/ prefix
    slug = post['slug']
    path = f"/blog/{slug}"

    # Extract keywords from code injection
    keywords = extract_keywords_from_code_injection(post.get('codeinjection_head'))

    # Extract og:image from code injection
    ogimage_url = extract_ogimage_from_code_injection(post.get('codeinjection_head'))

    # Build frontmatter
    frontmatter_data = {
        'published': is_published,
        'type': 'BLOG',  # Default to BLOG, can be manually changed
        'path': path,
        'date': published_at,
        'title': post['title'],
        'keywords': keywords,  # Extracted from code injection, empty if not present
        'topics': topics
    }

    # Add headerImageUrl if feature_image exists
    if post.get('feature_image'):
        frontmatter_data['headerImageUrl'] = post['feature_image']

    # Add ogImageUrl if extracted from code injection
    if ogimage_url:
        frontmatter_data['ogImageUrl'] = ogimage_url

    return frontmatter_data


def generate_filename(post: Dict) -> str:
    """Generate filename from Ghost post."""
    published_at = post.get('published_at') or post.get('created_at')
    date_str = published_at[:10]  # YYYY-MM-DD
    slug = post['slug']

    # If slug already has date prefix, use as-is
    if re.match(r'^\d{4}-\d{2}-\d{2}-', slug):
        return f"{slug}.md"

    return f"{date_str}-{slug}.md"


def sync_post_from_ghost(post: Dict, dry_run: bool = False) -> str:
    """Sync a single post from Ghost. Returns status: 'created', 'updated', or 'skipped'."""
    filename = generate_filename(post)
    file_path = CONTENT_DIR / filename

    # Convert HTML to Markdown
    markdown_content = convert_html_to_markdown(post['html'])

    # Download images (or check which would be downloaded in dry-run)
    download_images(post['html'], post['slug'], dry_run=dry_run)

    # Map to frontmatter
    frontmatter_data = map_ghost_post_to_frontmatter(post)

    # Create frontmatter post
    new_post = frontmatter.Post(markdown_content, **frontmatter_data)

    if dry_run:
        print(f"  [DRY RUN] Would write: {filename}")
        return 'skipped'

    # Check if file exists
    status = 'updated' if file_path.exists() else 'created'

    # Write file
    with open(file_path, 'w', encoding='utf-8') as f:
        f.write(frontmatter.dumps(new_post))

    return status


def main():
    """Main function to sync all posts from Ghost."""
    import argparse

    parser = argparse.ArgumentParser(description='Sync posts from Ghost CMS to Markdown')
    parser.add_argument('--dry-run', action='store_true', help='Preview changes without syncing')
    args = parser.parse_args()

    # Load environment variables
    load_dotenv(ENV_FILE)

    api_key = os.getenv('GHOST_ADMIN_API_KEY')
    if not api_key:
        print("Error: GHOST_ADMIN_API_KEY not found in .env file")
        sys.exit(1)

    # Initialize Ghost API client
    try:
        client = GhostAPIClient(GHOST_API_URL, api_key)
    except ValueError as e:
        print(f"Error: {e}")
        sys.exit(1)

    # Fetch all posts from Ghost
    print("Fetching posts from Ghost...")
    try:
        posts = client.get_all_posts()
    except Exception as e:
        print(f"Error fetching posts: {e}")
        sys.exit(1)

    total_posts = len(posts)

    if args.dry_run:
        print(f"\n[DRY RUN MODE] Processing {total_posts} posts...\n")
    else:
        print(f"\nProcessing {total_posts} posts...\n")

    # Track statistics
    stats = {'created': 0, 'updated': 0, 'skipped': 0}

    # Process each post
    for i, post in enumerate(posts, 1):
        print(f"[{i}/{total_posts}] {post['title']}")

        status = sync_post_from_ghost(post, dry_run=args.dry_run)
        stats[status] += 1

        if status == 'created':
            print(f"  ✓ Created")
        elif status == 'updated':
            print(f"  ✓ Updated")

    # Print summary
    print("\n" + "=" * 50)
    print("Reverse Sync Summary:")
    print(f"  Created: {stats['created']}")
    print(f"  Updated: {stats['updated']}")
    print(f"  Skipped: {stats['skipped']}")
    print(f"  Total: {total_posts}")
    print("=" * 50)


if __name__ == '__main__':
    main()

You can see this script does the following

• load all posts from the API

• convert HTML to Markdown while respecting certain rules for code blocks, etc.

• download images and media from posts into the expected static content directory

I ran this a couple times to verify the format would be as close to the existing source as possible (having 230+ examples to choose from definitely helps with testing) and then ran a full reverse sync once.

And again, what's better than doing this manually? Running it on a schedule!

name: Reverse Sync from Ghost CMS

on:
  schedule:
    # Run daily at 2 AM UTC
    - cron: '0 2 * * *'
  workflow_dispatch:
    inputs:
      dry_run:
        description: 'Run in dry-run mode'
        required: false
        default: 'false'
        type: choice
        options:
          - 'true'
          - 'false'

jobs:
  reverse-sync:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: pip install -r scripts/requirements.txt

      - name: Run reverse sync
        env:
          GHOST_ADMIN_API_KEY: ${{ secrets.GHOST_ADMIN_API_KEY }}
          PYTHONUNBUFFERED: 1
        run: |
          if [ "${{ github.event.inputs.dry_run }}" = "true" ]; then
            python scripts/sync-from-ghost.py --dry-run
          else
            python scripts/sync-from-ghost.py
          fi

      - name: Check for changes
        id: check_changes
        run: |
          if [ -n "$(git status --porcelain)" ]; then
            echo "has_changes=true" >> $GITHUB_OUTPUT
          else
            echo "has_changes=false" >> $GITHUB_OUTPUT
          fi

      - name: Count changes
        if: steps.check_changes.outputs.has_changes == 'true'
        id: count_changes
        run: |
          # Count new files
          NEW_FILES=$(git status --porcelain | grep "^??" | wc -l | xargs)
          # Count modified files
          MODIFIED_FILES=$(git status --porcelain | grep "^ M" | wc -l | xargs)

          echo "new_files=$NEW_FILES" >> $GITHUB_OUTPUT
          echo "modified_files=$MODIFIED_FILES" >> $GITHUB_OUTPUT

      - name: Create Pull Request
        if: steps.check_changes.outputs.has_changes == 'true'
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Configure git
          git config --global user.name "github-actions[bot]"
          git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"

          # Create branch with timestamp
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          BRANCH_NAME="ghost-sync-$TIMESTAMP"

          # Create and switch to new branch
          git checkout -b "$BRANCH_NAME"

          # Add all changes
          git add content/posts/ content/static/posts/

          # Commit changes
          git commit -m "Sync posts from Ghost CMS

          - New posts: ${{ steps.count_changes.outputs.new_files }}
          - Updated posts: ${{ steps.count_changes.outputs.modified_files }}

          Automated sync performed at $(date -u +"%Y-%m-%d %H:%M:%S UTC")"

          # Push branch
          git push origin "$BRANCH_NAME"

          # Create PR with detailed body
          gh pr create \
            --title "Sync posts from Ghost CMS ($TIMESTAMP)" \
            --body "## Summary
          This PR contains changes synced from Ghost CMS.

          - **New posts**: ${{ steps.count_changes.outputs.new_files }}
          - **Updated posts**: ${{ steps.count_changes.outputs.modified_files }}
          - **Sync time**: $(date -u +"%Y-%m-%d %H:%M:%S UTC")

          ## Changes

          The following content was synced from Ghost:
          - Markdown post files in \`content/posts/\`
          - Static images in \`content/static/posts/\`

          ## Review Checklist

          - [ ] Review new post content and frontmatter
          - [ ] Check image downloads completed successfully
          - [ ] Verify frontmatter mapping (especially \`type\` and \`keywords\` fields)
          - [ ] Ensure no sensitive content was synced
          - [ ] Test builds locally if significant changes

          ## Notes

          This is an automated sync. The \`type\` field defaults to \`'BLOG'\` and may need manual adjustment for guides/tutorials. The \`keywords\` field is empty and may benefit from manual population." \
            --head "$BRANCH_NAME" \
            --base main

      - name: No changes detected
        if: steps.check_changes.outputs.has_changes == 'false'
        run: |
          echo "No changes detected. Ghost CMS is in sync with repository."

I hope this post serves as inspiration if you've ever thought about setting up something similar. I can totally recommend using Claude Code for the tedious work here, so you can focus on your content!

I've had a journey over the last years of finding the values I care about most. In 2018, I chose to write my posts in Markdown to ensure they'd stand the test of time. Back then, MDX was all the rage, but the more components you could add, the more components you would have to maintain. I chose to limit my content to the lowest common denominator every software could agree on. In 2020, I moved to Next.js as I saw Gatsby nearing its end of life. GraphQL was very exciting for some time, and a rich pipeline of plugins was promising, yet it fell victim to complexity by trying to solve all use cases. Earlier this year, I moved from Vercel to my own server. And yesterday, I moved this blog from Next.js to Ghost, the last move in a series of steps that reinforced my commitment to owning my data and remaining independent.

Look, I don't think the world would end if I lost my content. It might be freeing to start from scratch, actually. But the latest string of vulnerabilities, as well as constant breaking changes convinced me it was time to settle on a boring yet reliable technology. I've learned the hard way that less is more. Maybe I'll host text files on a web server one day, and I'll be happy with it.

Another area I care about a lot is my collection of notes. I've been a happy Notion customer for five years. I admire the team's work in building one of the best products that helped me think and store my knowledge. And yet, I'm afraid of the day the company is acquired or the priorities change away from providing the best experience to seeking rent, as countless other companies have done in the past. Obsidian is an amazing alternative that may lack a feature or two but gets better every update. And the best part about it? Your notes are stored as Markdown, on your device.

We have the luxury of choosing the software we use. With every day that passes, it gets easier to choose an alternative that respects your choices. Luckily that doesn't mean you have to give up craftsmanship or polish anymore.

I work at the intersection of software engineering and management in startups, scaling engineering orgs from 0 to 1.

I write about my experience solving management and engineering problems in the context of startups. As I'm constantly learning new things along the way, topics will vary depending on what's most important to save for the future, as I'm creating a collection of resources of everything I think is relevant for building software and building and growing software companies.

That said, all timeless posts are tagged as Evergreen, for your convenience. And if you're interested in my software engineering origin story, you can read about it here.

Personal Favorites

• What's Next? (2022)

• Making Architects work in Software Teams (2022)

• Steady State means Continuous Rewriting (2022)

• Fundamental Design Decisions for Scalable Systems (2020)

At the intersection of engineering and management

In recent years, I've worked as a software engineer, building backend services handling millions of requests per day, using technologies like Golang, GraphQL, PostgreSQL, TypeScript, and Node.js. I've also spent a fair share of my time building frontend web applications using React.js, Apollo, and XState. In addition to full-stack web engineering, I'm also building mobile applications for iOS using Swift and SwiftUI/UIKit. While I love trying out new and experimental technologies, I strongly prefer battle-tested solutions and boring technologies for production systems.

Over time, I helped scale engineering teams, onboarding new engineers, making architectural decisions, and helping build a culture of trust and ownership to enable engineers to do their best work. I worked together with software engineers, product managers, leadership, sales, and marketing, aligning engineering efforts with business goals and making sure we build the things our customers need most.

What's my focus right now?

Since joining in May 2024, I've been building distributed systems at Inngest, helping engineering teams to focus on creating value and reducing complexity.

What do I value at work?

I am a builder with strong action bias. I value ownership, trust, and great communication in teams. I think that the best teams listen closely to their customers and build things people want, at the cutting edge of technology. While tech is great, I strongly believe that humans are the most important part of any organization, and that empathy, trust, and communication are the most important skills for any team to succeed. I love sharing my knowledge and experience with others, and I am always looking to learn from others, no matter their background or experience.

I am not limited to engineering: I enjoy working with people, enabling teams, and solving problems, regardless of the domain. I have worked on product, engineering, design, sales, and marketing challenges, and I am always looking to learn more about how to build great products and great teams. In the end, I really care about building great products that solve unmet needs, and I am happy to help out wherever I can.

Paying it forward

I have been fortunate to have many great mentors and teachers in my career, and I am always looking to pay it forward. If you're thinking about founding or already building a startup or if you are working on a project that you'd like to get feedback on, feel free to reach out to me via email and I'll do my best to help you out or point you in the right direction.

Colophon

This site is built with Next.js and TailwindCSS and hosted on Vercel. I am using Inter as sans-serif typeface and Spectral as serif typeface. The accent color is #0200ff.

I had been planning a project to replace a critical data pipeline system currently hosted on Google Cloud with Kafka, and I could have built it in a couple days, but I really wanted to test what's possible with Claude Code. So, instead of trying to blindly prompt along, I sat down for an hour and wrote a spec outlining a high-level summary, a detailed rollout plan, system reliability considerations, metrics to track, a definition of done, and step-by-step tasks to follow. The result was more an engineering spec than a prompt.

Feeding all this context to Claude, the worst outcome could have been completely broken, incomplete, or outright bad code. Even so, I would have been better off than before, as the spec in itself was invaluable: Writing down all context and constraints for Claude to follow forced me to think clearly. Steering and scoping, not writing code, represents the actual value-add. This is one of the key themes I'll try to emphasize in this post.

Back to the refactor, Claude took an hour to author all changes, and to my surprise, the results were near perfect. The changes I did add by hand were not clearly described in the specification, so I couldn't even blame Claude. Yet, even with some ambiguity, Claude managed to produce a reasonable first iteration for a non-trivial problem. It followed instructions on breaking down the problem into smaller parts, as well as using established patterns found in similar implementations across the codebase to ensure consistency. It repeatedly fixed issues to end up with code changes that successfully compiled and passed linting errors.

I'm surprised by how well this worked, and it got me thinking.

A short recap of recent history

ChatGPT was first released in late November 2022. At that point, LLMs had been around for some time (GPT-1 was released in 2018, GPT-3 in 2020) but weren't accessible to consumers, neither technically (no public products) nor in terms of usability (models weren't fine-tuned to handle instructions). ChatGPT delivered a dead-simple chat interface with a model that reacted to instructions (GPT-3.5), and it mostly just worked.

The first iteration of instruction-tuned LLMs felt revolutionary for generating and editing text, as well as helping with learning, but the lack of first-class tools and integrations created lots of friction. Interestingly, ReAct, the first big paper on reasoning and tool use was released way before reasoning models and function calling were integrated into frontier models. In February 2023, the Toolformer paper outlined how to equip LLMs with tools including a calculator to help with basic arithmetic and access to search engines to ground knowledge and prevent hallucinations.

A couple months after ChatGPT's initial release, GPT-4 was introduced in early 2023, with function calling APIs launching in June that year.

Early LLM products relied heavily on semantic search and context compression techniques like RAG to find relevant knowledge without exceeding the painfully limited context windows. Due to limited or missing function calling capabilities, models were often built to return encoded instructions to calling systems for orchestrating further steps.

With the launch of Cursor in March 2023, a new generation of AI-enabled developer tools started making waves. Fast-forward to late 2024 and reasoning models were announced by most AI labs. Similar to the approach of Chain-of-Thought (CoT) prompting, reasoning models produce reasoning or thought tokens at inference-time, which are then used to generate the final response, effectively applying more compute time to come up with a response. OpenAI published o1 in September 2024, Gemini 2.0 Flash Thinking arrived in December, and DeepSeek-R1 was released in January 2025. Claude 3.7 Sonnet was released in February 2025, featuring extended thinking as a more fine-grained control over reasoning capabilities. This current iteration of models is significantly better at selecting the right tool for a problem, enabling more autonomous or agentic experiences.

Newer models are sufficiently more capable in most dimensions: Gemini 2.5 Pro features a context window of 1m tokens, compared to the 32k context window originally supported by GPT-4, the flagship model released two years earlier. This enables applications to include vastly more data, grounding the model and yielding more relevant results. Reasoning capability allows the latest models to solve more complex problems in various disciplines and interact with tools much more reliably and effectively.

In this context, the first research preview for Claude Code was released on February 24, 2025, together with Claude 3.7 Sonnet. Just two months later, Anthropic announced the fourth generation of Claude models, and general availability for Claude Code.

UX matters

Claude Code embodies the state-of-the-art approach of equipping reasoning LLMs with tools to produce a smooth UX the same way early ChatGPT opened up LLMs to the public. While Cursor's Agent mode has been around since November 2024, something about using Claude Code in a terminal feels more ergonomic than an IDE+Agent combination.

For one, Claude is remarkably good at picking the right tools while hiding unnecessary noise from lookup operations. The agent loop works near-autonomously, yet you can always interrupt the model and update instructions when you see it getting off-track. And lastly, the IDE integrations are the perfect balance between sharing context/using IDE features like diff views without being too imposing on the conversation flow.

I've been using Claude Code for projects I've always wanted to do, yet never got around to doing. Tedious tasks like writing CI/CD pipelines or Infrastructure as Code or upgrading and migrating dependencies essentially disappear, so you don't waste time on tasks that don't add business value. Besides, having multiple tabs of Claude Code running in parallel feels like managing a small team, even if you can't actually step away (yet).

As with any new tool, there are rough edges. Resuming and forking conversations works out of the box, but with long threads I do fear that previous context influences subsequent responses too much, steering the model in the wrong direction. Restarting conversations from scratch isn't great either, so using project-level memory and docs like RFCs and ADRs may be a good compromise.

In the long run, I believe Claude Code will augment most internal tooling. There are tons of MCP Servers readily available and integrating the tools you use every day (issue tracking, observability) is pretty much a done deal. As models become cheaper and more powerful over time (better reasoning, even larger context windows, faster response times), tools like Claude Code will be able to write (author, test, review, maintain) better code, more autonomously.

If this sounds scary, let me bring up two important points.

Claude Code is incredibly good at writing code but your results will be subpar if you're not steering it well. Just as you wouldn't let an engineer work on a codebase without objectives, check-ins, or reviews, Claude can't read your mind (yet). While you may spend less time writing code with Claude, you will spend more time thinking about the code that needs to be written. And why. And when. Tools have always existed, and this certainly is the best toolbox on the market. Yet, what really matters are the decisions you make and the outcomes you achieve.

Second, real business value isn't in wrangling dependencies, figuring out how Terraform works, setting up an S3 bucket, writing GitHub Actions workflows, fixing linting issues, or refactoring a codebase. Real business value comes from interactions between people. It's created in understanding customer needs and building relationships. Real value is created by building teams and aligning people on shared goals. With tools like Claude Code, we have a lot more time for that. What a time to be a software engineer!

A note of caution

Claude Code and LLMs in general are very powerful tools and can solve writing tasks better than most people in a fraction of the time. They can support knowledge discovery and the learning process, but I strongly believe that you should not use them to avoid friction and difficulty in the first place, even if it's tempting, even if the results will be just as good or much better than doing it yourself. Here's why: Asking Claude to explain concepts like a codebase works incredibly well, but personally, I learn a lot from following a system end-to-end. While there are many kinds of people out there, I'm sure the following may sound familiar.

When I started writing my first pieces of software more than a decade ago, I spent hours and hours of my time producing bugs, and trial-and-erroring my way to a working piece of code. And while the resulting projects may not have been a big commercial success, they helped me learn. Every time I stumbled over some issue, I learned. Friction triggers awareness, which then drives the learning process.

Having an LLM apply learning tools like Socratic questioning ensures you don't accidentally let AI do all the work, which can be great. Yet, I wonder what academia and vocational training for computer science/software engineering will look like in a world where tools like Cursor and Claude Code are commonplace. Are people truly going to pick up concepts? Did people have the same questions when the internet and search engines came up? (Yes!)

Another risk of excessive AI use is that you have LLMs replace your truly important writing. I'm not talking about drafting some official-sounding letters, outreach message, website copy, or shitpost for Twitter. As engineers, we produce a lot of written content, both internally and externally. Writing helps me think clearly. Translating a problem in your head to a piece of paper in front of you forces you to communicate in a common language. You can use LLMs to proofread written content, but for the first draft, it's your turn.

To repeat, your value (as a software engineer specifically, and most related disciplines) is in identifying valuable problems to solve, weighing opportunity cost and deciding which problems to solve first, making necessary tradeoffs to scope problems into a realistic schedule, and delivering a solution. Just as a team lead delegates tasks, you're not measured by the lines of code you write but the outcome. So in a way, AI levels everyone up to a team lead of AI workers.

In this world, communication skills are more important than ever. Specifically, clear and concise writing. In addition, curiosity and willingness to try out new things. If you're not ready to learn and make some mistakes, it's infinitely harder to discover even better ways of working.

Wrapping Up

Claude Code, Cursor, and other AI products are very impressive and useful. While I can't gauge the full impact of progressively shifting to AI-driven software engineering, I can see the change in perception in my own ways of working. I can't wait to see how this space keeps evolving!

Early in 2024, I was working hard on building CodeTrail, a new take on documentation for engineering teams. During this time, I learned a lot about sales, marketing, and distribution. While Tim was working hard on building the product, I took over sales, creating cold outreach sequences, calling up potential customers, running demos, following up to leads — to make it short, I did everything under the sun to get our first customers on board. Unfortunately, due to a number of factors, we were bound to fail.

Neither me nor Tim had a large platform, both of us previously worked on high-growth startups but we underestimated the difficulty of activating our networks to discover the best pilot customers. We were solving a problem both of us experienced in our previous jobs, but in a market of layoffs and fears of an economic downturn, onboarding software understandably wasn’t on people’s minds. While we were able to craft a high-quality product, we were fighting an uphill battle in one of the most risk-averse markets on the planet.

After half a year, Tim and I decided to wind down our efforts on CodeTrail in order to reflect what we were missing to succeed and plan the next steps accordingly, removing any barriers to retry a couple years in the future.

While Tim is focusing on academia, I decided it made most sense to join a fast-moving team I could grow with. I spent March applying at and interviewing with 30+ companies in different growth stages and industries. At one point in my discovery stage, I got introduced to Inngest, a Bay Area/US startup solving the hard problem of building reliable software, a passion area of mine I spent the past years of my life working on. After the first interviews with the team, I was fully convinced that this was the best possible option out there.

I joined the engineering team at Inngest exactly 8 months ago in late April. Since then, we’ve increased our product usage nearly 20x, which introduced scaling challenges I had the opportunity to solve.

During my first weeks of onboarding, I built batch keys, a new feature to group event batches by a user-supplied expression. This is helpful in multi-tenant scenarios, for example to group in-app notifications for a given user, or creating massive mailing campaigns.

After my first project, I started focusing on systems and infrastructure, joining our founding engineer Jack Williams in implementing a critical new service for disaggregating access to our primary data store, and implementing a sharding strategy for horizontal scaling. These improvements enabled us to handle 100k+ QPS comfortably without a second of downtime. If you’re curious about the full story, you can read more on the Inngest blog.

After ensuring our infrastructure could handle the increased demand, I moved on to the core queueing systems, implementing a new architecture to improve multi-tenant fairness and throughput globally and for each account. Once this was rolled out, I spearheaded sharding efforts around the queue, unlocking horizontal scaling to handle a 5x increase in queue throughput.

I’ve learned a lot about designing, building, and operating distributed systems, performing gradual rollouts without degrading system availability, and engineering in high-growth environments. For the last months of 2024, I’ve been working on a major upcoming product and infrastructure feature, unlocking enterprise security, higher end-to-end throughput at lower latency, and a new execution model altogether. I’m excited to share more news on this soon!

Joining a fully-remote team also gave me the opportunity to meet everyone in person in Lisbon and San Francisco for company-wide and engineering offsites.

If you had told me what was about to happen throughout the year in January, I wouldn’t have believed you.

I’m forever grateful for the warm welcome by the Inngest team. I’m having an absolute blast working together with everyone and learning new things every day. Joining Inngest reignited my passion of engineering highly-scalable systems, making trade-offs to fit the growth stage, and building with a strong vision in mind.

Moving on to 2025, I’m excited to share big personal news relatively soon. Onward and upward.

— Bruno

It’s awkward to admit, but I actually believed Redis was just a key/value store. More specifically, I never imagined keys could be anything other than bytes, let alone powerful data structures. The second I started to read up on Redis, I changed my mind. After spending most of my working hours the past six months working with Redis, I want to present cases for and against using Redis, Valkey, or any related technology.

Before we get into this, let’s consider why we need to persist data, why it sometimes makes sense to choose a data store optimized for specific access patterns, and why Redis is a great match if you’re looking for a high-performance data structure store, both persistent and ephemeral.

Applications are rarely stateless

This probably won’t come as a shock, but it’s good to When you’re building a SaaS product or any piece of modern software really, you’ll need to persist state. This may be anything from data required for operations over user data all the way to data for bookkeeping like logs and history.

In old times, this state could have lived on a single machine, but nowadays, you’ll have to consider fault tolerance and chances are, you need to serve customers in multiple regions around the world. Simply put, one server storing all state in memory doesn’t cut it anymore.

There are many data stores, offering different durability guarantees, data modeling paradigms, scaling options, and other properties. And this is where it gets interesting: How do you choose a data store suitable for your application?

Data structures vs. relations

I grew up in a world of relational database management systems. Tables became my hammer for every problem. This is obviously silly: Relational databases have their use cases, but forcing data into a model it wasn’t made for will cause problems down the road.

Most notably, this means poor ergonomics and performance issues: Traditional relational databases are optimized for index-heavy access operations, carefully scanning as few rows as possible. Complex query planners are designed to optimize data access based on historical statistics. You can select index types, sure, but a lot of performance characteristics are predetermined and leave little control.

Redis isn’t opinionated on how you structure your data. Instead, it provides a wide range of in-memory data structures: Strings, Hashes (maps, dictionaries), Lists, Sets, Sorted Sets (my favorite), and more.

You might wonder why that’s exciting, aren’t those just the basic building blocks of any programming language? You’d be right and wrong: When you’re building applications for a single node, these data structures are omnipresent. Redis allows you to keep this simple data layout while allowing you to access and modify this state from any client, written in any language.

As simple as these data structures sound, they cover almost any use case: Want to implement a queue? Use a list or, if you’re fancy, a sorted set. Want to index data for exact matches? Use a hash for O(1) access. Locks? You’re covered.

If this sounds exciting, you may wonder if there are any downsides. That depends on your requirements. Like most decisions in engineering, there’s no one good or bad solution out there.

Choosing the right data store

Engineering is all about tradeoffs, and this applies to selecting the right data store. If we’re building a latency-sensitive production system, we can lay out some hard requirements:

• Availability: The system must withstand production workloads without downtime. This may require failover instances, multi-zone or multi-region replication, and most definitely

• Durability: Data must be persisted on disk. If the machine unexpectedly goes down, every stored record must be present when it becomes available again. To prevent losing data from missing writes during downtimes, see the Availability section for measures like automatic failover.

• Horizontal Scaling: Capacity planning is hard, and relying on a single instance can only buy so much time. Zero-downtime instance resizing is often impossible. Adding new nodes to the system must be easy and downtime during potential rebalancing operations must be kept to a minimum.

There are other requirements including reliable disaster recovery, the remaining ACID properties (atomicity, consistency, isolation).

Redis is interesting in that it offers atomic operations on a global keyspace, supports durability with snapshots and append-only files, and allows horizontal scaling through sharding the global keyspace across a distributed cluster. You can even write Lua scripts to support atomic operations across multiple Redis commands.

Yet, there are downsides: Redis is, at its core, single-threaded. And even though individual operations are really fast, you’ll inevitably run into CPU bottlenecks if you’re unable to distribute your data across shards. Horizontal scaling only works through sharding, but once you distribute your keys across different slots, you lose atomicity: Multi-slot operations are prohibited, and that includes Lua scripts.

I’m really excited about the release of Valkey 8: The Valkey maintainers have put in lots of effort to implement I/O multithreading while keeping execution single-threaded and sequential in nature, which boosts throughput and reduces latency across the board.

Redis and its derivatives are incredibly versatile data stores enabling applications both small and large in scale. If you’re aware of the pitfalls when it comes to scaling using clustering and sharding, you can get really far with this one component.

The search for a new challenge

After multiple attempts at building my own company, I decided it was time to switch things up. I was looking for a talented team focused on building a world-class product, preferably in the infra or DevTools space, preferably in the Bay Area. Around March, the landscape for engineering jobs wasn’t ideal, if you were aiming to get into BigTech. Fortunately, my favorite company environments start around product-market fit and scaling up.

Once I started interviewing with Darwin and the folks at Inngest, I immediately noticed the high energy and conviction with which they’re building a remarkable product. I ran some due diligence and checked on the competition, and even four months later our product is still the most powerful and easiest to learn out there. I was hooked, and luckily the team felt the same way, so I joined at the end of April.

The first months at Inngest

When I joined, I did multiple pairing sessions with Darwin and Tony to grasp the fundamentals of the engineering setup. In my second week, I started working on a critical project with Jack, the details of which have justified a blog post on the official company blog. After launching a set of infrastructure improvements to pool Redis connections in a dedicated service, we built, tested, and launched a sharding strategy within weeks. This was incredibly exciting and important for the infrastructure at the same time.

Fast forward to today, I’ve fully onboarded, worked on the core infrastructure, and already rolled out several mission-critical improvements which I’m very proud of. I’m also extremely grateful to everyone at Inngest for helping out wherever possible and sharing knowledge.

I’m writing this post on the plane back from Lisbon, where I just concluded my first offsite with the team. Working remotely can be hard because social connections and team bonding are hard to replicate in virtual meeting rooms. Having a week to get to know each other, tell stories, discuss current issues, and align everyone on the greater vision makes up for this. And it was an absolute blast.

The challenges ahead

Inngest, the product, has grown in usage by multiple orders of magnitude since I joined. Naturally, this presents a couple of challenges to the infrastructure we’ve been running for a while now. We’ve been hard at work designing and building the next version of many core infrastructure components to enable seamless growth for the next couple of years. We’ve made several key decisions that I’d love to share, but it will take some more time until we can unveil what we’re building.

At the same time, we’ll have to grow the team to free up capacity for building product features and other important work. We’re still in a stage where we operate with incredibly high autonomy, and taking ownership of areas of responsibility is expected.

I truly can’t wait for the months ahead!

This immense volume of data is stored in our primary Redis-based data store, presenting significant challenges in terms of throughput and latency due to Redis’ single-threaded design. Recently, our main cluster approached critical capacity limits, with peak CPU utilization and dwindling storage space threatening our system’s stability and performance.

Over the past six weeks, we implemented a series of infrastructure improvements that significantly reduced latency and improved long-term scalability. Impressively, we achieved this without a millisecond of downtime. In this post, I’ll give you an overview of the changes we’ve been working on, and focus on a new proxy service in the critical path.

To avoid degrading end-to-end latency and prevent storage capacity issues, we implemented a multi-stage plan focused on enhancing Inngest’s core components. Our first step was to introduce a unified state interface, consolidating all state operations in the critical path in a single place. This way, improvements are propagated across the entire system.

Next, we introduced a coordination service to decouple our services from direct connections to the primary data store. Instead, requests are routed through a lightweight proxy, which supports sharding and caching for better efficiency. Long-term, this architecture allows us to replace the current state store with a completely new backend optimized for Inngest’s access patterns and SLOs.

Let’s delve into the implementation of the state coordinator.

Implementing the State Coordinator

After defining the new state interface, we immediately started work on the new coordination service, routing all state interface requests to the current or future state store. We designed the interfaces with the new service in mind, so the handover was seamless.

Accessing the state store is a critical operation in our system because every function run involves reading and writing events, step outputs, and other metadata. Failure in these operations could result in significant performance degradation across the system.

We designed the state coordinator to offer high throughput at the lowest possible latency. We implemented resilient error handling by retrying transient errors introduced by the network roundtrip, as well as introducing observability on both the client and server side, giving us insights into service performance and potential issues as early as possible.

By having all services connect to very few state coordinator instances, Redis connections are pooled in a single place, making it easy to understand bottlenecks and scale the system accordingly.

The state coordinator was built with robust sharding and caching mechanisms, utilizing consistent or highest random weight (HRW) hashing. This approach ensures uniform data distribution and high cache hit ratios, by routing requests to instances most likely to store the data in question. Sharding can be achieved by applying the same logic on the client to determine which group of state coordinators serving a single state store cluster should be invoked. Adding a new shard is as easy as provisioning another cluster and gracefully reconfiguring existing services.

Rolling out

To ensure a smooth rollout, we devised a strategy to gradually enroll Inngest customers in using the state coordinator. We leveraged the state interface to create a rollout layer using feature flags, enabling controlled, percentage-based rollouts across various customer segments: Internal users, free users, paid customers, and finally, enterprise customers.

To mitigate the risk of unexpected errors, we implemented robust fallback logic, ensuring any failed requests are retried using the previous implementation. In the worst case, the customer wouldn’t even notice that our new service failed, and no data would be lost in the process.

These safeguards gave us the confidence to roll out the new code to production very quickly, as we could flip a switch and fall back within a second. After thoroughly testing the new service internally, we enrolled all non-enterprise customers within a week. During the entire rollout, we tracked rollout progress combined with operation success and latency metrics using a custom dashboard, which made it easy to catch any problems, such as slow or failing requests. In every way, the rollout started smoothly.

After continuing to evaluate our system performance and allocating more compute resources to proactively rule out bottlenecks, we enrolled all enterprise customers without a hitch.

The state coordinator has been running in production for over a month now. On average, we are serving 300k rpm, with a p99 of less than 20ms, including network roundtrip latency. This is a fraction of the traffic going to our primary data store, yet it accounts for the largest response sizes due to user-generated state outputs.

Post-Rollout

Through this project, I gained valuable insights into building and deploying critical infrastructure, designing for gradual customer enrollment, creating effective dashboards for monitoring system health, and effectively communicating project updates with the team.

After conducting the rollout and running the state coordinator in production, there are a couple improvements I’d look into the next time we run a similar project. First, percentage-based rollouts are an amazing tool to slowly ramp up usage to spot bottlenecks early. Simply flipping a switch and routing all production traffic to a new system without safeguards would be reckless at best.

Percentage-based rollouts get more predictable the more uniformly distributed your enrollment key is. This is similar to sharding where the wrong sharding key can cause a skewed data distribution. If you roll out based on account IDs, it’s fairly likely that you encounter a long tail of accounts with little to no usage, with a few accounts making up most of the activity in your system. All account IDs look the same to a rollout system, so it’s possible that you start seeing a huge spike after increasing your percentage by a microscopic amount.

Knowing your product usage metrics is crucial for understanding how choosing a certain enrollment key will affect the system. One solution for this problem could be to create multiple segments based on different historical usage and adjust the entities in the buckets over time, leading to higher predictability.

This has been my first major project to work on over at Inngest, and it’s been incredibly rewarding. Thanks to Jack, Darwin, Tony, and all the other folks on the team for helping out and making this possible!

When refactoring a production codebase, you don’t always have the budget or desire to rewrite the code consuming an interface. New implementations must match the previous logic down to the returned errors. Let’s see how this can be done when using gRPC.

Errors in Go

Go is built on conventions, including for handling errors. In Go, functions return errors alongside values. Errors can be wrapped and returned again, forming a chain of errors. This is quite nice as it helps you trace what went wrong, and where.

Below is an example demonstrating error chaining:

func getOneRow(sql string, args []any) (any, error) {
	res, err := db.Do(sql, args)
	if err != nil {
		return nil, fmt.Errorf("could not run command: %w", err)
	}
	return res, nil
}

func loadUser(userId string) (User, error) {
	user, err := accessDb("select * from user where userId = $1;", userId)
	if err != nil {
		return nil, fmt.Errorf("could not access database: %w", err)
	}
	return user, nil
}

func isAdmin(userId string) (bool, error) {
	user, err := loadUser
	if err != nil {
		return false, fmt.Errorf("could not load user: %w", err)
	}
	return user.isAdmin, nil
}

In some layer of the call hierarchy, internal errors are converted into business logic errors: Not finding a row in a database may be relevant for internal code, but a user cares about the fact that their account doesn’t exist because they’ve signed up with a different email address. User-facing errors can be added into the error chain and retrieved using errors.Is.

Errors over the network

All these conventions work as long as your code runs in a single process. When sending requests over a network using libraries like gRPC, traditional error handling conventions can be challenging due to differing error serialization methods. gRPC offers error codes and details for attaching rich information to errors, which is great when all consumers expect gRPC-style error handling. Sometimes, error chains are just converted to their string representation. Good luck checking a string for the existence of a specific error.

Depending on your use case, the fact that functions are invoked over the network shouldn’t be exposed to the downstream consumers of your APIs, though. When defining an interface like ImageCreator below, you specify expected behavior without detailing network-specific errors or concrete error types.

type ImageCreator interface {
	CreateImage(data CreateData) (Image, error)
}

type CreateData struct {
	Prompt string
}

type Image struct {
	URL url.URL
}

In other words, you can’t expect code consuming the ImageCreator interface to handle gRPC-specific errors.

Restoring Errors

Imagine we’re using the ImageCreator service outlined above to generate a header image for blog posts (how original).

func generateHeaderImage(creator ImageCreator) (url.URL, error) {
	img, err := creator.CreateImage(CreateData{"abstract blog post header"})
	if err != nil {
		if errors.Is(err, ErrInvalidPrompt) {
			showInvalidPromptDialog()
			return nil, nil
		}

		return nil, fmt.Errorf("something went wrong: %w", err)
	}

	return img.URL, nil
}

To inform a user about passing an invalid prompt, we define the following error

var ErrInvalidPrompt  = fmt.Errorf("invalid prompt")

Consider two concrete implementations: A local service running in the same process, and a remote version running on beefier machines, communicating via gRPC.

Both implementations can return this error, but it might be hidden in a chain of errors

could not create image: invalid prompt
could not send create image request: could not create image: invalid prompt

Both of these are completely valid, and we want to surface them to the user with a proper error dialog! By using errors.Is() in the client implementation, we can traverse the error chain until we find the quota exceeded error. For the server, it won’t be this easy. Remember that most implementations simply return a string! So what can we do here?

Most implementations use error codes for retaining application-specific context. A strategy involves detecting user-facing errors before returning them, transforming them into a network representation and back to a concrete error on the client.

Below is an example implementation of this strategy.

Common

var userFacingErrors := []error{
		ErrQuotaExceeded,
}

First, we define a slice of known errors. You could add any user-facing error here.

Server

func preProcessError(err error) error {
	for _, ufe := range userFacingErrors {
		if errors.Is(err, ufe) {
			return ufe
		}
	}

	return err
}

// CreateImage implements imagecreator.CreateImage
func (s *server) CreateImage(ctx context.Context, in *pb.CreateImageRequest) (*pb.CreateImageReply, error) {
	img, err := createImageOnServer(in)
	if err != nil {
		return nil, status.New(codes.InvalidArgument, preProcessError(err))
	}

	return &pb.CreateImageReply{Image: img}, nil
}

preProcessError will iterate over a range of all known user-facing errors and, if included in the given error, return only the known error. This will shorten error chains and return only a single known error.

Client

func restoreError(reason string) error {
	for _, err := range userFacingErrors {
		if err.Error() == reason {
			return err
		}
	}

	return nil
}

type imageCreatorClient {}

func (i *imageCreatorClient) CreateImage(data CreateData) (Image, error) {
	c := pb.NewGreeterClient(conn)

	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()
	r, err := c.CreateImage(ctx, &pb.CreateImageRequest{data})
	if err != nil {
		s := status.Convert(err)
		if restored := restoreError(s.Message()); restored != nil {
			return Image{}, restored
		}

		return Image{}, fmt.Errorf("something went wrong")
	}

	return r.Image, nil
}

The client extends the error handling approach started by preProcessError by verifying whether the message included in the gRPC status matches any known error. If this lines up, a proper Go error will be returned, which can then be checked by downstream consumers.

Restoring errors is an easy method to hide implementation details in scenarios where requests are sent over a network. This way, consumers can interact with interfaces as intended. Improved support for narrowing error types in future Go releases would be beneficial for building robust error handling. Until then, documentation is the best we can do to ensure our code behaves as expected.

The same rules apply in Go, take the classic example of io.Reader:

type Reader interface {
	Read(p []byte) (n int, err error)
}

This interface is used everywhere. It doesn’t matter if you’re reading from a file, a network socket, or a data structure in the same process: As long as you need to read bytes from a source, the interface can be used.

This is amazing because it allows decoupling the implementation from the downstream consumers. Your code shouldn’t care about how exactly a file is read from disk, or whether it is read from disk at all.

Interfaces enable many use cases, and in Go, they’re very idiomatic to use whenever it makes sense. One rather interesting case I’ve run into recently, was decoupling data access in our production codebase at Inngest.

An intro to Inngest

Inngest orchestrates durable functions and workflows, hosted on any service that can accept HTTP requests. Each function includes one or more steps triggered by events.

During a function run, we need to store details like the return value of your steps so we can load them on the next step. Here’s an example function that showcases Inngest’s capabilities: When a user signs up, Inngest runs steps to load the user data, send a welcome email, and wait for a post creation event, triggering further actions based on the outcome.

export default inngest.createFunction(
  { id: "activation-email" },
  { event: "app/user.created" },
  async ({ event, step }) => {
	  const user = await step.run("load-user", async () => {
      return await db.loadUser({ email: event.user.email });
    });

    await step.run("send-welcome-email", async () => {
      return await sendWelcomeEmail({ email: user.email, name: user.name });
    });

    // Wait for an "app/post.created" event
    const postCreated = await step.waitForEvent("wait-for-post-creation", {
      event: "app/post.created",
      match: "data.user.id", // the field "data.user.id" must match
      timeout: "24h", // wait at most 24 hours
    });

    if (!postCreated) {
      // If no post was created, send a reminder email
      await step.run("send-reminder-email", async () => {
        return await sendReminderEmail({
          email: event.user.email,
          name: user.name,
        });
      });
    }
  }
);

Storing Run State

To handle long-running functions, Inngest stores state details like loaded user data. With waitForEvent potentially delaying execution for extended periods, our system must manage state efficiently.

Previously, our execution logic was tightly coupled to Redis for storing function run state. As Inngest Cloud grew, we needed scalable solutions like sharding and caching. We unified state access behind a state.RunService interface, ensuring flexibility and scalability.

Consolidating Data Store Access

By creating new interfaces for state management, we created strong guarantees about the expected functionality that each implementation needed to follow. After merging this change, we started with the real work to prepare our system for the next stage.

To gracefully handle increasing system load, we needed to address two primary concerns:

• Too many connections to the backing data store

• Running out of storage capacity

High connection counts can strain systems like our highly-available data store used to persist function run state in our Cloud environment. This is a known problem. For example, Postgres creates a new backend process every time a connection is requested.

The solution here is to use a connection pooler like PgBouncer which acts as a proxy between your application and the Postgres instance. PgBouncer simply maintains a pool of opened connections, ready to be used for incoming client connections. Once all service connections are in use, new client connections need to wait until one server connection completes.

You might think “Why don’t you just use an application-level pooler?”, and that’s a great question! Instead of running one PgBouncer instance, you might initialize a connection pool in each application process, preventing it from establishing an unbounded number of connections. This works great as long as you limit the number of application processes. Once you create new services or auto-scale your production deployment, you might establish more connections than you’ve ever planned for. That’s why this approach is very limited in large, distributed systems.

To manage increasing connections to our data store and prevent storage issues, we introduced a coordinating service. This service, implementing the state interface, allowed us to handle connections efficiently and scale seamlessly.

Previously, replacing all state access logic with a new implementation would have taken weeks, not to mention the risk we would have taken in potentially missing one or two locations in our codebase.

With the new interface, we simply swapped the direct connection logic for the coordinator service in our production environment. Admittedly, we added more precautions for rolling out to all customers, but the idea was the same.

Interfaces saved the day.

Thanks a ton to Darwin, Jack, and Tony at Inngest for supporting me on this effort, it’s been super exciting to future-proof one of the most important pieces of our infrastructure.

If you’re looking for ways to build internal tools and workflows or want to implement multi-tenant queuing, please give Inngest a try!

After a couple of months, you’ve shipped the initial version of this workflow and requirements have changed. Unfortunately, the worker processing the messages needs to be updated to handle a slightly different format.

You’ve thought long and hard and decided to prepare the following rollout plan: For a short transition period, you’ll ensure that the system keeps processing both old and new messages. This is done by updating the consumer to handle both the new and old message formats. The producer is updated to provide the new format. After all old messages have been processed, you can remove the code handling the old format.

You prepare a pull request including updates to both the producer and consumer and all tests pass. You feel good about handling the problem and deploy to production. You’re using a managed container service and watch while new instances are spinning up. Suddenly, your error rate starts spiking. Messages are failing in the consumer. You begin to sweat a little and start exploring the current logs to find consumer containers complaining about malformed messages. Of course, that’s why you added graceful handling in the pull request! You assume that the new code is broken and decide to roll back. For some reason, the errors keep coming in.

What’s going on? Let’s rewind.

Orchestration systems like Kubernetes usually deploy new containers side by side with old instances. This allows for a graceful, zero-downtime rollout. This also means that two versions of your code are running side by side.

As distributed systems are notoriously hard to predict, there’s no way that you can or should be timing the deployment of different components in one deployment. That’s why you can’t be certain that all consumers are deployed before producers start rolling out.

Let’s go back to the time of deploying your changes.

After sifting through the errors some more, you notice that all logs originate from old containers. Suddenly, everything starts to make sense!

You roll forward to the latest version, wait for a couple of minutes until all old pods have terminated, and the errors disappear.

Let’s recap what went wrong here. First, you expected your changes to roll out in a certain order. With most systems, you do not get these guarantees. Second, you did not handle the case where old consumers were picking up messages enqueued by new producers. You have to be really really careful that new consumers handle old messages and old consumers handle new messages. If either invariant is violated, you’re potentially breaking your system.

So how could this have been avoided?

One possible solution could have been to separate the deployment of producers and consumers. You could have split up your pull request to isolate the changes enabling graceful handling of old messages to the consumer. After deploying this and waiting for all old consumers to terminate, you could have followed up with another deployment updating the producer code to publish new messages. At that point in time, no new messages could have physically reached old consumers.

In any case, the system should have gracefully retried the failing messages. After a couple of attempts, a message should have reached a new consumer and succeeded, even if the initial attempt failed. As long as you never drop a message that wasn’t processed yet, the system is able to recover (albeit after some downtime).

Rolling out changes in distributed systems is a tricky problem because your tests won’t catch this kind of inconsistency. Tests run on the current version of the codebase, which compiles just fine. Ideally, your test runner would know about old code running in production and test producer changes against old and new builds.

Instead of writing all of this yourself, you should be using a solution like Inngest, which happens to be my current employer.

Shipping new features, improvements, or other changes can require careful planning ahead of a release. While you can try to imagine and rule out different failure scenarios ahead of time, you never really know what’s going to happen until you ship.

Sure, you could set up shadow reads/writes or split production traffic to hit an isolated clone running the new code, but depending on your resource budget and current infrastructure, that effort might be hard to justify.

For new versions of an existing feature, you could also fall back to the old logic on failure or compare new output with existing results and log any deviations, as they occur.

In small, fast-moving teams, you want to roll out features at a faster pace, but you cannot afford to break the experience for higher-paying and enterprise customers. So what if you could gradually roll out new features across segments, starting with free users and moving up the subscription tiers in steps?

Feature flags are a tool used for toggling specific parts of your product on or off for a certain set of users without having to reboot or reconfigure services. With feature flags, you can distinguish between different segments (groups of users) and gradually enable a flag for a small and then growing percentage of users over time. If something goes wrong, you can simply disable the flag and users will work on the previous version.

This flow only works when you can switch between implementations without migrations. In case you’re releasing a big feature by migrating individual users, consider enrolling eligible accounts in the migration process incrementally. You should probably strive to limit large migrations to speed up the rollout process and reduce your team’s maintenance burden.

In case you’re switching between two implementations with identical interfaces, feature flags are a perfect match. Slowly increase the percentage of users exposed to the new implementation and optionally enable fallbacks to the previous version for all users. Once everyone is using the new code, you can slowly remove the fallbacks as long as the new implementation doesn’t cause any issues. Once nobody is using the fallback, you can completely drop the old code.

In the past, I’ve used LaunchDarkly to create and integrate feature flags into products I worked on. With LaunchDarkly, you can configure feature flags to target specific users in different segments (or a percentage thereof), without worrying about performance degradation, as flag rules are stored in memory and evaluated without a network roundtrip.

Properly planning feature rollouts removes a lot of stress factors from the team. If you have a clear plan and options to roll back to the previous experience if anything goes wrong, your coworkers and customers will thank you for it.

In November 1987, RFC 1034 and RFC 1035 first defined the domain name system (DNS), a distributed database containing the names of resources, designed to be extensible beyond the initial scope. Software including browsers and other internet-enabled apps refers to a DNS resolver, which queries name servers to resolve hostnames to IP addresses, mail settings, or other information.

To facilitate communication between DNS resolvers and name servers, RFC 1035 specifies the DNS wire format. DNS messages are transmitted as a series of octets (bytes). In this guide, we’ll try to understand what goes into a typical DNS query and response by implementing a DNS message parser in Go.

 0               1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       1       |       2       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       3       |       4       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       5       |       6       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Before we get to parsing messages, let’s look at some fundamental concepts used through DNS, including domain names and resource records.

Domain Names

<domain> ::= <subdomain> | " "

<subdomain> ::= <label> | <subdomain> "." <label>

<label> ::= <letter> [ [ <ldh-str> ] <let-dig> ]

<ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str>

<let-dig-hyp> ::= <let-dig> | "-"

<let-dig> ::= <letter> | <digit>

<letter> ::= any one of the 52 alphabetic characters A through Z in
upper case and a through z in lower case

<digit> ::= any one of the ten digits 0 through 9

Domain Names as defined in Section 2.3.1 and Section 3.1. are represented as sequences of labels separated by dots. Labels must start with letters followed by optional numbers and can contain, but not end with hyphens. Each label is represented as a one-octet length field followed by that number of octets.

Every domain name ends with the null label of the root (visualized using a trailing dot), which is why a domain name is terminated by a zero-length octet.

To accommodate for message compression, the first two high-order bits of every label’s length octet must be zero. In case compression is used, both bits are 1, followed by an offset value pointing to another label or domain name.

func parseDomainName(fullMessage []byte, bytes []byte) (string, []byte) {
	domainName := ""
	for {
		length := bytes[0]

		isCompressed := length&0b1100_0000 > 0
		if isCompressed {
			pointer := (bytes[0] << 2) + bytes[1]

			resolvedPointer, _ := parseDomainName(fullMessage, fullMessage[pointer:])
			domainName += resolvedPointer

			bytes = bytes[2:]

			break
		}

		bytes = bytes[1:]

		if length == 0 {
			break
		}

		label := bytes[0:length]

		domainName += string(label)
		domainName += "."

		bytes = bytes[length:]
	}

	return domainName, bytes
}

Resource Records (RRs)

As I’ve said before, the domain name system was created as a database to translate resource names to information like IP addresses and other values. The atomic unit in this database is referred to as a Resource Record (RR) and is defined in Section 3.2.1. Each RR is addressed using a name (as defined above) and stores resource data (RDATA) for a given resource type (TYPE) and class (CLASS), expiring after the TTL.

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                                               /
    /                      NAME                     /
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     CLASS                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TTL                      |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   RDLENGTH                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
    /                     RDATA                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Resource Records exist for different data types like IPv4 (A records) and IPv6 (defined in RFC 3596) addresses, as well as text (TXT) and mail-related (MX) records.

type RDataCNAME struct {
	CName string // <domain-name> which specifies canonical or primary name for owner, owner name is alias
}

func (d RDataCNAME) String() string {
	return d.CName
}

type RDataMX struct {
	Preference uint16 // lower values are preferred
	Exchange   string // <domain-name> of host willing to act as mail exchange
}

func (d RDataMX) String() string {
	return fmt.Sprintf("%d %s", d.Preference, d.Exchange)
}

type RDataNS struct {
	NsdName string // <domain-name> specifying host which should be authoritative for specified class
}

func (d RDataNS) String() string {
	return d.NsdName
}

type RDataTXT struct {
	TxtData string // one or more character strings
}

func (d RDataTXT) String() string {
	return d.TxtData
}

type RDataA struct {
	Address string // host internet address
}

func (d RDataA) String() string {
	return d.Address
}

type RDataAAAA struct { // RFC 3596
	Address string // 128-bit IPv6 address
}

func (d RDataAAAA) String() string {
	return d.Address
}

type ResourceRecord struct {
	Name     string       // domain name associated with record
	Type     RecordType   // two octets containing RR TYPE code
	Class    RecordClass  // two octets containing RR class code
	TTL      uint32       // time in seconds until record should be refreshed in cache
	RDLength uint16       // length of octets in RData
	RData    fmt.Stringer // variable-length string of octets describing resource, format varies by TYPE and CLASS
}

// transparently handles empty sections
func parseResourceRecords(fullMessage []byte, messageBytes []byte, numRecords uint16) ([]ResourceRecord, []byte, error) {
	resourceRecords := make([]ResourceRecord, numRecords)

	for i := range resourceRecords {

		var domainName string
		domainName, messageBytes = parseDomainName(fullMessage, messageBytes)

		resourceRecords[i].Name = domainName

		resourceRecords[i].Type = RecordType(binary.BigEndian.Uint16(messageBytes[0:2]))
		messageBytes = messageBytes[2:]

		resourceRecords[i].Class = RecordClass(binary.BigEndian.Uint16(messageBytes[0:2]))
		messageBytes = messageBytes[2:]

		resourceRecords[i].TTL = binary.BigEndian.Uint32(messageBytes[0:4])
		messageBytes = messageBytes[4:]

		resourceRecords[i].RDLength = binary.BigEndian.Uint16(messageBytes[0:2])
		messageBytes = messageBytes[2:]

		switch resourceRecords[i].Type {
		case TypeA:
			// read first 32bit/4 byte
			ipv4 := net.IPAddr{
				IP: messageBytes[0:4],
			}
			messageBytes = messageBytes[4:]

			resourceRecords[i].RData = RDataA{
				Address: ipv4.String(),
			}
		case TypeAAAA:
			// read first 128bit (https://datatracker.ietf.org/doc/html/rfc3596#section-2.2)
			ipv6 := net.IPAddr{
				IP: messageBytes[0:16],
			}
			messageBytes = messageBytes[16:]

			resourceRecords[i].RData = RDataAAAA{
				Address: ipv6.String(),
			}
		case TypeCNAME:
			var canonicalDomainName string
			canonicalDomainName, messageBytes = parseDomainName(fullMessage, messageBytes)

			resourceRecords[i].RData = RDataCNAME{
				CName: canonicalDomainName,
			}
		case TypeMX:
			rDataStr := string(messageBytes[0:resourceRecords[i].RDLength])
			messageBytes = messageBytes[resourceRecords[i].RDLength:]

			preference := binary.BigEndian.Uint16([]byte(rDataStr)[0:2])
			var exchangeDomainName string
			exchangeDomainName, messageBytes = parseDomainName(fullMessage, []byte(rDataStr[2:]))

			resourceRecords[i].RData = RDataMX{
				Preference: preference,
				Exchange:   exchangeDomainName,
			}
		case TypeNS:
			rDataStr := string(messageBytes[0:resourceRecords[i].RDLength])
			messageBytes = messageBytes[resourceRecords[i].RDLength:]

			var nsDomainName string
			nsDomainName, messageBytes = parseDomainName(fullMessage, []byte(rDataStr))

			resourceRecords[i].RData = RDataNS{
				NsdName: nsDomainName,
			}
		case TypeTXT:
			rDataStr := string(messageBytes[0:resourceRecords[i].RDLength])
			messageBytes = messageBytes[resourceRecords[i].RDLength:]

			resourceRecords[i].RData = RDataTXT{
				TxtData: rDataStr,
			}
		}
	}

	return resourceRecords, messageBytes, nil
}

On its own, resource records are not very useful: We want clients to be able to query DNS name servers to retrieve relevant records by specifying a domain name, record class, and type. To transmit queries like this, the RFC introduces a common format for messages.

Messages

    +---------------------+
    |        Header       |
    +---------------------+
    |       Question      | the question for the name server
    +---------------------+
    |        Answer       | RRs answering the question
    +---------------------+
    |      Authority      | RRs pointing toward an authority
    +---------------------+
    |      Additional     | RRs holding additional information
    +---------------------+

Both queries and responses are transmitted using messages, as defined in Section 4.1. Messages start with a header, followed by one or more questions, resource records for answers, pointing to authoritative name servers (authority), and related information (additional) which are not strictly answers to the question.

type Message struct {
	Header     MessageHeader    // always present
	Question   []QuestionEntry  // question for name server
	Answer     []ResourceRecord // RRs answering question
	Authority  []ResourceRecord // RRs pointing towards authority
	Additional []ResourceRecord // RRs holding additional info
}

To parse the entire message, we must start with the header and work our way through the bytes.

func parseMessage(messageBytes []byte) (Message, error) {
	fullMessage := messageBytes

	headerBytes := messageBytes[0:headerSizeBytes]
	header, err := parseMessageHeader(headerBytes)
	if err != nil {
		return Message{}, fmt.Errorf("could not parse header: %w", err)
	}

	messageBytes = messageBytes[headerSizeBytes:]
	questionEntries, messageBytes, err := parseQuestions(fullMessage, messageBytes, header.QDCount)
	if err != nil {
		return Message{}, fmt.Errorf("could not parse questions: %w", err)
	}

	answerRRs, messageBytes, err := parseResourceRecords(fullMessage, messageBytes, header.ANCount)
	if err != nil {
		return Message{}, fmt.Errorf("could not parse answer RRs: %w", err)
	}

	authorityRRs, messageBytes, err := parseResourceRecords(fullMessage, messageBytes, header.NSCount)
	if err != nil {
		return Message{}, fmt.Errorf("could not parse authority RRs: %w", err)
	}

	additionalRRs, messageBytes, err := parseResourceRecords(fullMessage, messageBytes, header.ARCount)
	if err != nil {
		return Message{}, fmt.Errorf("could not parse additional RRs: %w", err)
	}

	return Message{
		Header:     header,
		Question:   questionEntries,
		Answer:     answerRRs,
		Authority:  authorityRRs,
		Additional: additionalRRs,
	}, nil
}

Let’s start by parsing the message header!

Header

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      ID                       |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    QDCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ANCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    NSCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ARCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

The header section is defined in Section 4.1.1. of RFC 1035 and includes important information to identify and match the current response to a client request, determine whether the current message is a request or response (QR), specify a query kind (OPCODE), response code (RCODE), and other important flags.

type MessageHeader struct {
	ID      uint16   // identifier assigned by program that generated query, copied to reply
	QR      bool     // one-bit field that specifies if message is query (0, false) or response (1, true)
	OPCode  OpCode   // kind of query
	AA      bool     // only in response, specifies that responding server is authority for domain name (corresponds to name matching query or first owner name in answer)
	TC      bool     // TrunCation, specifies message was truncated due to length greater than permitted on the transmission channel
	RD      bool     // Recursion Desired: set in query, copied to response, directs name server to pursue query recursively
	RA      bool     // Recursion Available, set (1) or cleared (0) in response, denotes whether name server supports recursive queries
	Z       struct{} // reserved, must be 0
	RCode   RCode    // response code
	QDCount uint16   // number of entries in questions section
	ANCount uint16   // number of RRs in answer section
	NSCount uint16   // number of RRs in authority records section
	ARCount uint16   // number of RRs in additional records section
}

From the specification, we already know the header to be 2 octets * 6 rows = 96 bits = 12 bytes in length. If this doesn’t match up, we can return an error.

With some bitwise operators, we can extract the relevant values from our structure, shifting bits whenever we need to access misaligned values in the octets. Using the encoding/binary package, we can translate byte sequences to numbers, which is used for parsing the ID and count values.

const headerSizeBytes = (8 * 2 * 6) / 8 // two octets * 6

func parseMessageHeader(headerBytes []byte) (MessageHeader, error) {
	if len(headerBytes) != 12 {
		return MessageHeader{}, fmt.Errorf("expected header to be 12 bytes")
	}

	id := binary.BigEndian.Uint16(headerBytes[0:2]) // first two bytes = 16 bits = 2 octets

	secondRow := headerBytes[2:4] // second two bytes = 16 bits = 2 octets

	qr := secondRow[0]&byte(0b1000_0000) > 0

	// after consuming QR, shift entire octet to the left so that opcode takes up leading bits
	secondRow[0] = secondRow[0] << 1

	// retrieve four-bit opcode and shift right by remaining 4 bits in octet to "index" at 0 instead of 8
	opcode := OpCode(secondRow[0] & byte(0b1111_0000) >> 4)

	aa := secondRow[0]&byte(0b0000_1000) > 0
	tc := secondRow[0]&byte(0b0000_0100) > 0
	rd := secondRow[0]&byte(0b0000_0010) > 0

	ra := secondRow[1]&byte(0b1000_0000) > 0

	rcode := RCode(secondRow[1] & byte(0b0000_1111))

	qdcount := binary.BigEndian.Uint16(headerBytes[4:6])
	ancount := binary.BigEndian.Uint16(headerBytes[6:8])
	nscount := binary.BigEndian.Uint16(headerBytes[8:10])
	arcount := binary.BigEndian.Uint16(headerBytes[10:12])

	return MessageHeader{
		ID:      id,
		QR:      qr,
		OPCode:  opcode,
		AA:      aa,
		TC:      tc,
		RD:      rd,
		RA:      ra,
		Z:       struct{}{},
		RCode:   rcode,
		QDCount: qdcount,
		ANCount: ancount,
		NSCount: nscount,
		ARCount: arcount,
	}, nil
}

Once we have parsed the message header, we know how many questions, and resource records we can expect.

Question

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                     QNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QCLASS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Questions are defined in Section 4.1.2. and consist of a domain name (QNAME), a query type (QTYPE), and class (QCLASS).

func parseQuestions(fullMessage []byte, messageBytes []byte, numQuestions uint16) ([]QuestionEntry, []byte, error) {
	questionEntries := make([]QuestionEntry, numQuestions)

	for i := range questionEntries {
		var domainName string
		domainName, messageBytes = parseDomainName(fullMessage, messageBytes)
		questionEntries[i].QName = domainName

		questionEntries[i].QType = QType(binary.BigEndian.Uint16(messageBytes[0:2]))
		messageBytes = messageBytes[2:]

		questionEntries[i].QClass = QClass(binary.BigEndian.Uint16(messageBytes[0:2]))
		messageBytes = messageBytes[2:]
	}

	return questionEntries, messageBytes, nil
}

This is all it takes to build a DNS message parser! Admittedly, it doesn’t handle cool extensions like DNSSEC, but you could go ahead and extend it easily. This goes to show that the original idea of an extensible naming system has lived up to its promises, fulfilling a critical role in powering internet infrastructure for decades to come.

I was pleasantly surprised how straightforward it was to follow the RFC and implement the parser in Go. If I find the time, I might look at other RFCs and try to implement more protocols. If you have any suggestions or questions, feel free to send a mail!

In the coming decades, there’s only going to be more software to build, operate, and scale for platforms we’re only just inventing, with code running on nearly everything we interact with day to day.

To help advance the current frontier into AI, which is built on massively distributed computing, understanding the engineering and technology stack from end to end has never been as important as today.

I’ve always had a deep interest in understanding how businesses and technology work at the foundational level. That’s why I joined the team at Hygraph when they were 5 people building out of a small office. I took the opportunity to learn the software engineering stack for massively multi-tenant B2B SaaS products. I helped build globally-distributed and highly-available cloud infrastructure, and I got to understand what it takes to scale an engineering organization from Pre-Seed through Series B, taking care of hiring and onboarding processes, documentation, and developer experience among other areas.

At TUM, I got insights into computer science and business fundamentals, getting into operating systems and computer architecture, distributed systems and heterogeneous computing with FPGAs, as well as corporate finance, business law, and operations research. I found it invaluable to discover areas that I wouldn’t have explored on my own.

After completing my studies and having built a couple of side projects over the years, I ramped up efforts in starting my own company together with my long-time friend, Tim. We believed the timing was spot on, and we wanted to solve problems we experienced ourselves in past jobs. First, we attempted to validate Anzu, which provided essential building blocks for engineering teams wanting to move fast, followed up by CodeTrail for providing better onboarding documentation.

We grew and got better every iteration, especially in sales and marketing. We crafted well-designed products, created strongly-personalized sales sequences, and ran cold-calling sessions to get our first customers.

Unfortunately, the cards were stacked against us. We failed to identify and validate an urgent, unmet business need. We didn’t have the network to collaborate with pilot customers before writing a line of code. And we didn’t have the capital for continued exploration into other ideas.

So we quit.

After a long and candid discussion with Tim, we decided it made sense for us to go back to square one and grow both our skills and network, our credentials, and financial independence with the eventual goal of getting back to building a business. The next time we work together as founders, however, we’ll solve a real problem with real customers and the capital to sustain the grind until we reach exit velocity.

Thus, I accelerated the pace of interviewing and talking to different teams in March, seeing where I could best bring in my experience and have an outsized impact and growth opportunities. Through this process, I learned a ton already, but that’s for another time.

Today, I’m excited to announce that I’m joining Inngest as a Senior Distributed Systems Engineer starting next Monday.

In the past, I’ve spent more time than I’d like to admit on decoupling critical services, managing distributed queues, handling failing jobs gracefully, and wrapping my head around distributed transactions. Inngest makes it ridiculously easy to build durable functions and background jobs, without managing any of the infrastructure. It’s almost magical.

Earlier this year, the team raised $6.1M in new funding by a16z (Andreessen Horowitz), with follow-on investment from existing investors: GGV, Afore Capital and Guillermo Rauch. I’m joining Inngest to help scale the backend infrastructure for the next growth stage and beyond.

Over the past weeks, I’ve had an exceptionally great interview process and incredible support from Tony and the entire team at Inngest on all process-related questions. I’m absolutely thrilled to get to know everyone on the team over the coming weeks and dive into the engineering process once again.

I’ll keep you posted 👋

A heartfelt thank you to everyone who gave their advice and recommendations over the course of this long-winding process and throughout the lows and highs, this wouldn’t have been possible without you.

Previously

Long-time readers may recall previous iterations of this blog that included an Algolia-powered search experience. When I switched out Gatsby in favor of Next.js for the foundation of the current stack in 2020, I removed all the bells and whistles I didn’t want to maintain anymore, including the in-house newsletter and search solutions. I wanted to focus on the essence of the blog, the content.

Over time, discoverability has become a challenge: New readers have a hard time finding content they’re curious about, and I’m losing track of what I wrote about years ago. Introducing a new search experience is one of the first steps I’m taking to improve content discoverability, with more changes to follow in the coming weeks.

Getting to Hybrid Search

When I started exploring the possible ways to implement search, I decided not to run with a managed solution like Algolia. Instead, I want to build a search stack tailored to the blog’s needs.

When users search for content, they might remember some detail of a post they read, or they might be interested in a certain technology. They might have come across one of my startups or projects. They might enter an area of work like system architecture, continuous integration, or containers. And they may paraphrase, use synonyms, or other related words.

In short, they will use a thousand possible ways to reach the same goal, finding a post. And I need to supply the right tools to make this task easier.

Keyword or full-text search allows finding exact matches, for instance, returning all posts mentioning CodeTrail, my most recent startup. This breaks down when you diverge from the phrasing I’ve used. Replace VM with virtualization and you’re out of luck.

Semantic search operates on learned representations of concepts in vector space. Imagine having to group similar words in a room. You might put ice closer to cold, and coffee closer to hot and warm. Containers are close to virtualization and far away from surfboard. So-called embedding models are trained to group words in an n-dimensional space minimizing the space between tokens that should be semantically similar. Comparing the embedded vector representation of a query string with the embedded vector representation of a blog post’s content using cosine similarity ranks closely-matching (approaching 1) or unrelated (approaching 0) results. Intuitively, this works because minimizing the angle in each learned dimension of the vector space corresponds with close semantic similarity.

Semantic search is a powerful tool to fetch semantically similar results for a query and the same result for semantically similar queries at the same time.

Semantic search operates on a predefined vocabulary that might not know about the latest technology. Running “Apple Vision Pro” through an embedding model trained before the last WWDC will most likely not arrange it closely to “virtual reality”. This is why semantic search should be coupled with a more traditional search strategy respecting keywords.

Hybrid search approaches combine semantic search with keyword or full-text search to get the best of both worlds. Reciprocal Rank Fusion (RRF) is an algorithm that combines two ranked result lists into one unified result set. In our implementation, we will run semantic search and full-text search separately and use RRF to create a single list of results.

Choosing the stack

The current blog iteration is built with Next.js, and deployed on Vercel. Every post is a Markdown file, assets like images are uploaded and distributed separately. Markdown is processed using the unified ecosystem of tools including remark and rehype and finally converted to React components and rendered in the browser.

To store full-text search and embedding vectors, I decided to create a PostgreSQL instance running Supabase with the pgvector extension enabled to store vector columns and perform semantic search using cosine similarity operators between query and embedding vectors.

To convert natural language search queries to a suitable representation in vector space, I’m using the OpenAI Embeddings endpoint with the recently launched text-embedding-3-small model.

I’ve built an indexer service running on every push using a GitHub Actions workflow. All search requests are sent to a Vercel Edge Function endpoint. I use PostHog to understand how the search experience is adopted.

Most of the decisions are pretty standard for a small MVP-stage project and will serve for a while. I’ve deliberately planned this feature as an experiment to improve discoverability issues, and I might replace it with a better solution in the future.

End-to-End Implementation

To populate the posts database with new or updated post content, I’ve created a GitHub Actions workflow running an index script on push or manual dispatch.

The indexer loads all Markdown post content from disk, extracts frontmatter, and iterates over each post. It creates a full-text search vector incorporating post title, slug, keywords, and other metadata. This procedure can be changed in the future.

To prevent generating vector embeddings for unchanged posts, we calculate a hash of the current file content and compare it to the current post entry in the database. If the post hasn’t been indexed previously or hashes don’t match, we create vector embeddings for the post.

To support longer content, we split the Markdown content into chunks using langchain’s MarkdownTextSplitter. We then create embeddings for each chunk using OpenAI’s text-embedding-3-small model with just 512 dimensions compared to the legacy text-embedding-ada-002's 1536 dimensions while achieving better performance.

Finally, we insert post embeddings into the database using the vector column data type provided by pgvector.

At runtime, when a user enters a search query, Next.js sends a client-side GET request to the search endpoint deployed using Vercel Edge Functions. We try to cache responses to the same query to reduce load, if possible. The function generates vector embeddings for the query string to calculate cosine similarity later on. We also generate a full-text search vector using websearch_to_tsquery.

Finally, we invoke the hybrid search routine, which separately finds the best matches for semantic and full-text search and applies RRF to calculate a unified result set. We also apply user filters on keywords, publishing time, etc.

Moving to Production

To provide a predictable experience, I’ve instrumented both frontend and backend, adding PostHog to analyze search usage for future improvements. I’ve also added service quotas to prevent adversarial users from exhausting system resources and configured billing limits in third-party services to prevent unexpected overspending.

Iterations

Initially, I created a minimal search experience using only semantic search. This worked well for semantically similar queries but returned suboptimal results when adding little-known entity names. Adding keyword search and combining results using RRF solved this issue.

To help users find specific content more easily, I enabled filtering support on topics, which are used throughout the blog to tag areas of work, programming languages, tools, and more.

In the future, I’ll probably add more filters and tune hybrid search parameters to perform better. This should make it even easier to sift through all posts to find the needle in the haystack.

Thanks for reading all the way through this! This project has been an interesting challenge, building a usable search experience takes more effort than I initially expected. The first public release is now live, feel free to try it by clicking the search button on the top right or hitting ⌘K.

As it turns out, building rich-text editors is surprisingly complex. From modeling the underlying data structures used for storing formatted content to creating an intuitive UX that supports a natural editing experience, there’s a ton of heavy lifting that makes a well-designed editor feel simple.

Previously, implementations used various workarounds and hacks, but with the introduction of the contenteditable attribute, browsers enabled a new class of RichText editors using the DOM to render and manipulate content.

ProseMirror is a low-level set of building blocks to construct RichText editor experiences. Think of it as a thin abstraction layer on top of contenteditable HTML, taking care of schema enforcement, state management, and content rendering.

ProseMirror powers popular libraries like TipTap (which is used by other libraries like Novel and BlockNote). I believe a big part of its popularity comes from being intentionally unopinionated to create a solid foundation for customizable editing experiences.

In most cases, I recommend using a wrapper like TipTap simply because it’s easier to work with, but sometimes, you really need all the power. So let’s dive into the basic concepts behind ProseMirror.

Useful resources

Before we get started, I’ve collected some helpful resources that make working with ProseMirror infinitely easier.

First, the ProseMirror Developer Tools Chrome extension allows you to view the internal state of any ProseMirror instance detected on a webpage, including the document content (in a tree-like view, more on that later) with its history, plugins, and schema.

When you want to understand specific parts of ProseMirror, the official guide is an amazing starting point. The examples section also shows what you can build with ProseMirror, including Markdown support, collaborative editing, and text linting.

The ProseMirror Document Model

ProseMirror is organized around documents to represent content. A document contains block nodes. A block node may be a paragraph, image, list, or other element. Instead of nesting content in a tree structure, however, ProseMirror attempts to lift up inline text content as far as possible, leading to a very shallow structure.

Imagine we’re writing the following text:

<p>Hello <b>World</b></p>

You may expect the document to look like the following:

- node: paragraph
  - text: Hello
  - node: bold
    - text: World

This is not what ProseMirror does, however.

- node: paragraph
	- text: Hello
	- text: World (bold)

Marks like bold, emphasis, inline code, links, and more are not nested but attached to the text.

Flattening the data structure allows ProseMirror to use positioning within a paragraph for more natural text selection and editing operations without tree manipulation. Furthermore, each document has only one valid representation:

Adjacent text nodes with the same set of marks are always combined together, and empty text nodes are not allowed. The order in which marks appear is specified by the schema, too.

If you’re venturing into the true depths of ProseMirror, nodes have different property combinations:

A typical "paragraph" node will be a textblock, whereas a blockquote might be a block element whose content consists of other blocks. Text, hard breaks, and inline images are inline leaf nodes, and a horizontal rule node would be an example of a block leaf node.

Defining the Document Schema

Now that we’ve learned how documents are constructed, let’s continue with schemas. If you’re building a product like Notion, you might want to allow your documents or pages to include specific blocks like paragraphs, images, code, and more.

ProseMirror determines whether a document is valid by checking against a predefined schema. This is defined similarly to parser grammar, as you get to define which nodes exist in the system and what they may contain:

const trivialSchema = new Schema({
  nodes: {
    doc: {content: "paragraph+"},
    paragraph: {content: "text*"},
    text: {inline: true},
    /* ... and so on */
  }
})

In the example snippet above, a document may include one or more paragraphs, while a paragraph may include zero or more inline text nodes. Schemas can represent more complex data models like Markdown. If you want to add custom node types like interactive blocks, you need to register the nodes in your schema first.

In addition to handling validation, the schema also defines how ProseMirror renders your document in the DOM and how DOM content is parsed back into a node (relevant for enabling copy-and-paste support).

Creating an initial state

Whenever you create an editor instance, ProseMirror has to keep track of the current content, selection, plugins, and other data, all of which is wrapped up in the editor state. Creating the initial state for ProseMirror to use is as simple as

import {schema} from "prosemirror-schema-basic"
import {EditorState} from "prosemirror-state"

let state = EditorState.create({schema})

If you’re editing an existing document, you can pass the previous document right into the create call.

Rendering the editor

You could use ProseMirror completely headless and create your own view layer on top, but luckily, you don’t have to. Create an instance of EditorView by passing the DOM element to render the editor in as well as the initial state and ProseMirror will render the current document.

let view = new EditorView(document.body, {state})

This works great when you’re not running in some other framework like React. Let’s explore how we can wire up ProseMirror to work together with React.

Using React

When you’re building a React application, you’ll likely want to use React state management primitives like hooks. You might pass the current document content into the editor and expect updates to be passed back up. The NY Times has published a ProseMirror wrapper library for React that bridges the two worlds nicely.

To render the ProseMirror editor within React, simply render the <ProseMirror/> component:

import { ProseMirror } from "@nytimes/react-prosemirror";

const initialState = ...

export function ProseMirrorEditor() {
  const [mount, setMount] = useState<HTMLElement | null>(null);

  return (
    <ProseMirror mount={mount} defaultState={initialState}>
      <div ref={setMount} />
    </ProseMirror>
  );
}

If you want to lift up the state and manage it yourself, that works too:

export function ProseMirrorEditor() {
  const [mount, setMount] = useState<HTMLElement | null>(null);
  const [state, setState] = useState(EditorState.create({ schema }));

  return (
    <ProseMirror
      mount={mount}
      state={state}
      dispatchTransaction={(tr) => {
        setState((s) => s.apply(tr));
      }}
    >
      <div ref={setMount} />
    </ProseMirror>
  );
}

Internally, the <ProseMirror/> component will create, mount, and manage a ProseMirror EditorView instance through useEditorView.

Bonus: Node Views

I hear you: All this is great, but can we customize nodes like paragraphs using React? As always, there is more than one solution. You could change the styles using CSS for the entire editor. But if you need to use anything from the React component tree (e.g. user preferences), you’ll need something different. Unfortunately, the editor view breaks out of the React paradigm: Everything up until the ProseMirror component lives within React, but the EditorView directly manipulates the DOM.

Fortunately, node views allow you to take back control of the rendering process for a node. And with react-prosemirror, node views render React components using Portals.

Let’s say we want to apply TailwindCSS classes to every paragraph node. For this, we need to create a node view

import {
  useNodeViews,
  useEditorEventCallback,
  NodeViewComponentProps,
  react,
} from "@nytimes/react-prosemirror";
import { EditorState } from "prosemirror-state";
import { schema } from "prosemirror-schema-basic";

// Paragraph is more or less a normal React component, taking and rendering
// its children. The actual children will be constructed by ProseMirror and
// passed in here. Take a look at the NodeViewComponentProps type to
// see what other props will be passed to NodeView components.
function Paragraph({ children }: NodeViewComponentProps) {
  const onClick = useEditorEventCallback((view) => view.dispatch(whatever));
  return <p onClick={onClick}>{children}</p>;
}

// Make sure that your ReactNodeViews are defined outside of
// your component, or are properly memoized. ProseMirror will
// teardown and rebuild all NodeViews if the nodeView prop is
// updated, leading to unbounded recursion if this object doesn't
// have a stable reference.
const reactNodeViews = {
  paragraph: () => ({
    component: Paragraph,
    // We render the Paragraph component itself into a div element
    dom: document.createElement("div"),
    // We render the paragraph node's ProseMirror contents into
    // a span, which will be passed as children to the Paragraph
    // component.
    contentDOM: document.createElement("span"),
  }),
};

This isn’t the classic node view that ProseMirror exposes, instead, we’re rendering a React component. Next, we need to register our custom node views in the editor

const state = EditorState.create({
  schema,
  // You must add the react plugin if you use
  // the useNodeViews or useNodePos hook.
  plugins: [react()],
});

function ProseMirrorEditor() {
  const { nodeViews, renderNodeViews } = useNodeViews(reactNodeViews);
  const [mount, setMount] = useState<HTMLElement | null>(null);

  return (
    <ProseMirror mount={mount} nodeViews={nodeViews} defaultState={state}>
      <div ref={setMount} />
      {renderNodeViews()}
    </ProseMirror>
  );
}

This way, we can use ProseMirror for a natural rich-text editing experience and add our custom components back into the document. It’s the best of both worlds!

ProseMirror is incredibly powerful and can be used for building best-in-class editors on the web. At CodeTrail, we’re building our core documentation experience using ProseMirror, and I’ll dive deeper into some practical applications in upcoming posts!

With some debugging, I quickly tracked down the issue to the loading and preprocessing step. Before running any data analysis, the duration should be converted to milliseconds, as that’s easier to work with and plot later on.

machine,duration
1,0.045
2,0.035
3,0.358

As you can see below, the original author used the built-in CSV reader to stream the data into Python. As a hint, the first mistake I made was not looking into how the data is parsed by Python, because the code does look pretty straightforward.

from csv import reader

rows = []
with open('./test.csv', 'r') as csv_file:
    csv_reader = reader(csv_file)
    next(csv_reader)  # Skip the header row
    for row in csv_reader:
        rows.append(row)

# convert duration to seconds
rows = [(row[0], row[1] * 1000) for row in rows]

Running the conversion step blew up the memory usage from around 5 GB to over 130 GB. Unfortunately, I had to terminate the script early because my Mac was swapping extensively and everything started to break down past 100 GB.

Obviously, something was wrong. I reduced the data set to 10 rows to debug every step without waiting forever. Inspecting the layout of the rows quickly yielded an interesting piece of information: Instead of floating-point data types, the duration (and all other rows for that matter) were read as strings. So rather than efficiently storing the values as 32-bit or 64-bit floats, Python’s CSV reader didn’t know how to parse the duration without any additional details and opted for strings which offer more possibilities but are the worst possible option for numerical values.

Multiplying a string with a number in Python does not (in classic JavaScript fashion) convert the value to a number and run with it like below. See for yourself below:

$ python3
>>> x = "0.045"
>>> x * 10
'0.0450.0450.0450.0450.0450.0450.0450.0450.0450.045'
>>> x = 0.045
>>> x * 10
0.44999999999999996
>>>

$ node
Welcome to Node.js v20.9.0.
> "0.5" * 10
5

Ah well, this is wrong on so many levels. Let’s start and fix it.

If we wanted to keep this script working as is and don’t introduce any additional libraries, we should convert the data types as early as possible to prevent running into these issues.

rows = []
with open('./test.csv', 'r') as csv_file:
    csv_reader = reader(csv_file)
    next(csv_reader)  # Skip the header row
    for row in csv_reader:
        row[1] = float(row[1]) # convert strings to floats
        rows.append(row)

Adding one line to the input reader will convert the duration to proper floats. This simple change brought memory usage down back to around 5 GB and allowed the script to run to completion.

Alternatively, we could have used a library like pandas to create a data frame from the CSV immediately:

import pandas as pd
df = pd.read_csv('./test.csv')
df['duration'] = df['duration'] * 1000

Not only is this much simpler, it’s way harder to screw up.

You might think “This will never happen to me” and “Oh boy, those were some seriously stupid mistakes”, but I wonder how many codebases end up in a situation like this. The root cause was manifold:

First, a lack of context blocked me from spotting the mistakes: I usually write JavaScript so the Python way of converting between data types wasn’t on my mind. I also used Pandas and other libraries in the past and expected the built-in CSV handling to parse input values similarly.

Second, the lack of a strict type system and validation layer made it harder to spot unexpected differences in data types. It might be helpful to require users to define their column types ahead of time and raise an exception if unexpected input values are supplied.

Finally, unit tests on smaller but representative data sets would have surfaced the invalid implementation at the time of writing the code.

As a software engineer, I’ve been following the formation of a vibrant ecosystem of projects around LLMs. As a co-founder of Gradients & Grit, I’ve explored production-ready ways of integrating LLMs into existing products.

A lot of progress has been made in the open-source model landscape. Yet, before building system-critical functionality on top of LLMs, teams need to address some underlying issues. In the following, I’ll explore the core areas holding back LLM adoption in production environments.

Interpretability, Reliability, Consistency

In the past, it was easy to design systems and trace outputs back to inputs. You could evaluate systems with requirements and compare different implementations. With machine learning and especially deep learning, the connection between inputs and outputs has gotten fuzzy by design. Distributed systems sit at every level, adding randomness and non-determinism to the mix.

With LLMs, differences in architecture, training data, inference, and parameters supplied in each request can lead to wildly different responses, without accounting for hallucinations. When you’re consuming a model through an API, there’s no guarantee that you receive a matching response for the same request twice in a row.

This creates complexity throughout the entire software engineering lifecycle, from development to production. For starters, it’s hard to nail the right prompt (one metric could be the best performance for the lowest token usage) for a use case. This prompt isn’t portable as you can’t switch the underlying model and expect similar results. You can’t even expect the same model to yield the same results in the future. With open-source models, it should be easier to pin your system to a specific model version, which is great for consistency. Similar to regular dependencies, you’ll have to update or at least re-visit prompts once you update the model.

If you’re self-hosting the model, you’ll need to worry about serving it in production, and handling inference on a cluster of machines. This is distributed computing with near real-time requirements, which has been a challenge for every other use case in the past. If you’re opting for a managed solution, you can offload some complexity at the cost of giving up control and your data. Later on, we’ll see that the latter is a relevant blocker for enterprise customers and government institutions.

Of course, in traditional software engineering, you had to spend time to get to a good solution while balancing trade-offs, too. Yet, the degree of fuzziness or lack of guarantees with LLMs and machine learning in general has to be accounted for. You need to be aware of the system’s biases leading to unexpected consequences for a wrong prediction. This is why most products have incorporated AI in nice-to-have features with human supervision. Running LLMs in the background without any safeguards may be an expensive mistake.

This is not to say that traditional software isn’t prone to bugs that can destroy livelihoods, it’s just harder to test notoriously unpredictable systems.

Another interesting facet of working with LLMs is that you’re switching between strict and fuzzy logic when crossing boundaries between code and prompts. LLMs accept a modicum of inconsistency when it comes to formatting prompts, so chaining prompts works better than expected. Factual inaccuracy in one response might lead to a game of telephone when passed along a chain of LLMs, though.

Unfortunately, it’s hard to evaluate correctness once you serialize the result back into code. Forcing models to output a well-defined format like JSON has shown to help, and support for steerability is getting better with newer models.

Cost-Effectiveness

Let’s assume we’ve found a suitable use case for Generative AI and created a prototype for it. How expensive will it be to run this in production? Since OpenAI is subsidized by Microsoft, they can afford to operate at a loss throughout the near future, but unless you’ve raised a comfortable round recently, you might need to run a tight ship.

Running your own hardware is expensive. Hosted inference solutions scale with demand, which is great. Tokens can be cheap or expensive, based on the model’s capabilities. Some tasks need complex reasoning, while others can be done with less. Some prompts need a lot of details, some don’t. Figuring out the right balance requires a certain playfulness early on in the development process. Balancing different model characteristics can determine how feasible the business plan is down the road.

It’s important to understand the costs you will incur at different levels of scale. If it’s impossible to achieve positive unit economics, you’ll have a hard time in the best possible scenario. The bigger the role AI plays in delivering your product, the more you need to match your pricing. Traditional compute resources have become commoditized and allowed operating on incredibly high margins. Quantization and other efficiency improvements for models will eventually drive resource prices down.

Operational Knowledge

All AI-enabled products I’ve recently worked on sounded simple in principle and required a surprising amount of hardening and preparing for production readiness before launching. Knowing the necessary measures to run and scale a system is as important as designing the product itself.

It’s still early enough for operational knowledge to be built and iterated on for the first couple of times. There are few absolute truths and most teams are learning on a daily basis. Some voices criticized that prompt engineering is often lacking the engineering part.

As with any new paradigm, teams will slowly try out Generative AI and iterate until they’re confident about delivering a high-quality experience to their customers. It may make sense to build internal tools first and start in the background, then expand to adding AI features to the product. Then, some companies are working at the cutting edge and building their entire business on top of Generative AI.

I’m convinced that the community will share their knowledge and work together so that every team can eventually benefit from the progress made in the biggest companies.

Tooling

While it’s easy to put together a demo application consuming an API, building production-ready AI-enabled products requires many building blocks. For the same reasons that operational knowledge is still being created, solid tools for building LLM applications are yet to be created.

From tailoring prompts to analyzing model usage, tracking costs, monitoring errors, and evaluating consistency, the entire LLM lifecycle needs to be covered by the LLMOps movement. It’s already really easy to consume open-source models through APIs and hosted inference endpoints. Frameworks like Haystack and langchain make it easier to bring your own data, and I imagine more will follow in the months to come.

Enterprise: Safety and Data Governance

Depending on your customers, offering a product isn’t as straightforward as plugging into the OpenAI APIs. When you need to guarantee that customer data never leaves the jurisdiction, you’ll have to resort to regional service providers or resort to hosting models yourself. The latter is quite expensive, but if you’re aiming for government contracts, trust is an important currency.

The easier it becomes to run models with a smaller resource footprint, the easier it gets to run inference at the edge where it’s needed. Moving all components closer together also decreases latency, so customers will enjoy a snappier experience.

Most current service providers are US-focused and don’t sufficiently cover other geographic areas and jurisdictions. There’s a reason why Germany is seeing a ton of startups offering the same services available abroad for a GDPR-conscious enterprise sector. This may not be the best moat, but it’s certainly a market waiting to be served.

I’m hardly the first person who’s been thinking about running Generative AI in production. There’s a lot of hype and people expect LLMs to play a huge role in the years to come. To make this happen, we need to ensure that systems are reliable, consistent, cost-effective, and safe. Coming up with features to solve problems is one part of the solution, but proper execution requires solid software engineering. LLMs aren’t just used by researchers in labs anymore and the ecosystem has to adapt to this new reality.