JavaScript

Most teams don't decide to build microservices. They get pushed into it. One app grows for a couple of years. More people push into the same codebase. Then a change to something totally unrelated breaks checkout on a Tuesday. Nobody planned that. That's usually when someone says it, half-joking, half not: maybe we should just split this thing up. And Node.js is the name that comes up. Not because anyone ran a deep framework comparison. Honestly, half the time it's already running the API layer and chewing through small request/response calls all day, so nobody has to fight for it. It's already there. Easiest sell in the room. What people get wrong going in: the win isn't "we use Node.js now." It's narrower than that. Node.js microservices earn their keep when a service actually needs to scale on its own — checkout during a flash sale, say, while the blog section sits idle. Split things up without that need, and you haven't built microservices. You've built one tightly coupled app, just now with network calls between the pieces instead of function calls. Same mess. Slower. The real work is designing the microservices architecture in Node.js properly: keeping services loosely coupled, getting them to talk without one outage taking three other services down with it, and figuring out which pieces genuinely need their own database versus which ones are fine sharing. That's what this covers. Core Architecture Components A Node.js microservices architecture usually has the same handful of pieces, even if the specifics change from one company to the next. componentpurposecommon tools API Gateway Routes requests, handles auth, rate limiting Express Gateway, Kong, NGINX Service Framework Builds individual business services Express, Moleculer Synchronous Calls Request/response between services axios, fetch, gRPC Async Messaging Event-based communication RabbitMQ, Kafka Resiliency Prevents cascading failures Opossum (circuit breaker) Containerization Isolates services and dependencies Docker Orchestration Scaling, restarts, rollouts Kubernetes Logging Centralized, searchable logs Winston, Pino Monitoring Tracks performance and health Prometheus, Grafana 1. API Gateway Clients never talk to your services directly. They hit the gateway first, and it figures out where the request needs to go. This is usually also where auth checks happen and where rate limiting lives, so one client can't flood the system with requests. 2. Individual Services Behind the gateway are the actual services, each one handling a single piece of the business: orders, users, whatever it is. Express is still the default choice for building these. Some teams are moving to Moleculer instead, since it's built specifically for microservices rather than being a general framework stretched to fit. When choosing a Node.js microservices framework, the right option depends on how much infrastructure your team wants the framework to handle. 3. Database Per Service This is the corner teams cut, and it always shows up later, usually a few months in, once nobody remembers why the shortcut got taken. If the order service and the user service are both querying the same database, you don't actually have two services. You have one database wearing two name tags. Each service needs to own its data, full stop. Need something from another service? Ask through its API, or listen for the event it fires. Don't go around the back and query its tables directly; that's the shortcut that turns into a rewrite. 4. Message Broker Not every interaction needs an answer right away. When someone places an order, the order service shouldn't sit around waiting for a confirmation email to go out; it fires off an event and moves on to the next request. Something else, usually RabbitMQ or Kafka, is listening for that event and deals with it on its own time. How to Build Microservices With Node.js The honest answer to how to build microservices with Node.js is: don't start by spinning up five repos. Start by figuring out where the actual boundaries are. Each service needs to own one business capability, its data, its logic, everything it needs to run without leaning on another service to function. A practical Node.js microservices tutorial usually comes down to a sequence like this: Define service boundaries: Figure out the independent business functions: users, orders, payments, notifications, whatever they are for you.Create a Node.js project for each service: Deployable on its own, with its own dependencies and config. Not a shared node_modules folder pretending to be independent.Choose the right framework: Express is fine for lightweight services; reach for a dedicated Node.js microservices framework such as Moleculer when you need more built-in.Expose APIs: Give each service a clean REST or gRPC interface for anything synchronous. This approach keeps building microservices with Node.js focused on business boundaries rather than simply splitting a large codebase into smaller applications. Node.js Microservices Example A simple Node.js microservices example could be an e-commerce application divided into four services: User service: Manages customer accounts and authentication.Product service: Handles product information and inventory.Order service: Creates and tracks customer orders.Notification service: Sends email or other order-related notifications. For example, when a customer places an order, the Order Service can publish an order.created event. The Notification Service listens for that event and sends the confirmation without forcing the Order Service to wait for the email process to finish. This is also a practical example of how to create microservices in Node.js: start with independent business capabilities, expose only the interfaces other services need, and use events when a response isn't required immediately. Communication Strategies Some requests need an answer right away. Others just need to notify another service that something happened, and nobody's waiting on a response. Most Node.js microservices setups use a mix of both. Synchronous Calls One service asks, waits, gets an answer back. That's really it. Most of the time plain HTTP is enough: axios or fetch, nothing fancy. gRPC only earns its keep once two services are hammering each other with requests constantly and the JSON overhead starts showing up in your latency numbers. It runs over HTTP/2, uses Protocol Buffers, and has smaller payloads. Asynchronous Messaging Different situation. Order comes in; the order service doesn't need to hang around until the confirmation email actually sends; it just says done and picks up the next request. Somebody else deals with the email later. RabbitMQ if you care about routing, sending different messages down different paths. Kafka if you're dealing with volume, logs, activity streams, stuff that never really stops flowing. Design Patterns for Resiliency Distributed systems fail in ways a single app never does. One service going down shouldn't mean the whole system goes down with it, so a few patterns exist specifically to contain that damage. Circuit Breaker Something's failing, so the instinct is to retry, and retrying just adds load to a service that's already drowning. A circuit breaker cuts that off. After enough failures in a row, it stops sending requests to that service for a stretch and lets it recover instead of burying it further. Opossum is what most people reach for in Node.js when they're setting this up. Saga Pattern You can't roll back a transaction across three different databases the way you'd roll back one. So instead of a single transaction, you get a chain; each step commits on its own in its own service. If step four fails, you don't just stop; you run backward through one, two, and three, undoing what already happened. It's not clean. It's what you're left with once one database per service is no longer optional. Idempotent Consumers Networks resend messages sometimes; that's just how it goes. If your order service can't tell a retry apart from a brand new order, you end up double-charging someone eventually. A uniqueness check on the event solves most of this; before acting on a message, the service checks whether it's already seen it. Dead Letter Queues Some messages are never going to process no matter how many times you retry them: bad data, a broken payload, whatever the cause. Rather than let one bad message jam everything behind it, it gets pulled into its own queue and dealt with separately later, instead of stalling the rest of the line. Production Deployment and Observability Getting this running locally is one thing. Running it in production with actual traffic is where most of these decisions get tested for real. Containerization Each service, along with its database and anything else it depends on, gets wrapped in its own Docker container. This keeps one service's dependencies from clashing with another's, and it means what runs on your laptop is basically the same thing that runs in production, no more "works on my machine." Orchestration By the time you've got more than two or three containers, doing this manually just doesn't hold up. Kubernetes takes that off your plate; more traffic comes in, it spins up more instances on its own. Something crashes, it gets restarted without anyone needing to notice at 3 am. Rolling out a new version doesn't mean downtime either; it shifts traffic over gradually. And on the security side, secrets management means your API keys aren't just sitting in a config file somewhere waiting to get committed to git by accident. Centralized Logging Logging to a file on each individual server doesn't work once you've got a dozen services running across different machines. Nobody's going to SSH into ten boxes trying to piece together what happened. Tools like Winston or Pino send structured logs somewhere central instead, so you can actually search across everything at once when something breaks. Metrics and Monitoring The goal is finding out something's wrong before a user emails you about it. In a Node.js system specifically, event loop lag is the one to watch closely; a blocked event loop doesn't throw an error, it just quietly slows everything down until someone notices things feel off. Memory usage and response times matter too, obviously. Prometheus is usually what's pulling these numbers together, and Grafana is where you'd actually go look at them. Wrapping Up None of this is complicated on its own: gateway, services, a message broker, some way to keep failures from spreading. What makes it hard is doing all of it at once, correctly, while the system is already handling real traffic and you don't get a do-over if you get the database boundaries wrong on day one. Node.js fits well here mostly because it doesn't get in the way. It's lightweight, it handles the kind of request volume microservices tend to generate, and the ecosystem around it — Express, gRPC libraries, message broker clients — is mature enough that you're not building plumbing from scratch. Whether you're pulling a monolith apart piece by piece or starting fresh, the patterns covered here (separate databases, circuit breakers, idempotent consumers, proper observability) are the parts that actually determine whether the system holds up once it's under load, not just when it's running clean on your laptop.

Let's be honest about what happens when you give an entire engineering team AI coding assistants. You look at the sprint board, and tickets are moving to "In Review" faster than ever. Your developers are happy. They are writing boilerplate in seconds and generating entire component structures before their morning coffee gets cold. If you measure productivity by the sheer volume of code generated, your team has successfully turned into a factory. But then you look at your deployment frequency. It's flat. Depending on the week, it might actually be trending downward. How can a team be writing code twice as fast, but shipping to production at the exact same speed? Image 1: The AI Productivity Illusion The problem is that output increases, but throughput/deployments remains static or drops. The challenge is how to address this. The answer lies in basic systems engineering. If you optimize a step in a process that isn't the primary bottleneck, your overall throughput doesn't change. We just spent the last two years making typing faster. But typing was never the hardest part of software development. As engineering leaders, we are flying blind if we rely on legacy productivity metrics in an AI-assisted world. The bottleneck hasn't disappeared; it has simply shifted downstream. To actually measure and manage engineering velocity today, you need to abandon "lines of code" and start tracking the new friction points. Here are the three metrics you need to start watching immediately. 1. PR Cycle Time (and the "Rubber-Stamp" Ratio) AI tools are incredible at generating massive blocks of code. They are decidedly not incredible at explaining the architectural reasoning behind why they generated it. When a developer uses an LLM to build a feature, they often submit a massive Pull Request. For the author, it took ten minutes. For the senior engineer assigned to review it, it's a nightmare. Reviewing 500 lines of AI-generated code in a GitHub diff requires significantly more cognitive load than reviewing human-written code, because AI code often lacks a recognizable, human train of thought. The immediate result? Your PRs sit in the queue for days. What to track: Time to First Review / Time to Merge: If this metric is spiking while ticket completion is dropping, your bottleneck is purely code review. You need to enforce strict PR size limits. AI or not, a PR should rarely exceed 300-400 lines of logic. The Rubber-Stamp Ratio: Look for massive PRs that are approved in under five minutes with a simple "LGTM." This means your senior engineers are overwhelmed and are just pushing AI code through without reading it. That is a ticking time bomb for production outages. 2. CI/CD Pipeline Stability and P95 Build Times More code means more tests. AI is perfectly happy to generate fifty unit tests for a single utility function. On the surface, high test coverage looks great to upper management. But if those tests are poorly constructed, overly coupled to implementation details, or reliant on flaky assertions (especially in UI testing), your CI/CD pipeline is going to grind to a halt. We’ve all seen it: a developer merges a feature, and the build fails three times in a row because of a flaky, auto-generated E2E test that didn't properly await a DOM element. What to track: Build Failure Rate (specifically on the main branch): If your developers are moving fast but your pipeline is constantly turning red, your velocity is zero. P95 Build Duration: Is your pipeline taking 45 minutes to run because it's executing thousands of low-value, AI-generated tests? You might need to implement a "Build Gardener" rotation—assigning an engineer each sprint to aggressively prune useless tests, update assertions, and optimize pipeline caching. 3. The Code Churn Rate and Architectural Decay Think of an AI coding assistant as a highly enthusiastic, incredibly fast junior developer. It solves the immediate problem right in front of it. It does not look at the holistic architecture of your application and decide to refactor a base class for long-term scalability. If developers lean too heavily on AI for problem-solving, you end up with massive amounts of duplicated logic and hyper-localized fixes. The code works today, but the architecture rots tomorrow. What to track: Code Churn (Percentage of code rewritten within 3-4 weeks of being merged): High churn in an AI environment usually means developers are using trial-and-error via prompts rather than thinking through the system design. They merge a feature, realize it breaks an edge case, and have the AI write a messy patch the very next week. Tech Debt Ratio vs. Feature Delivery: You have to force developers to slow down and do architectural planning before they open their IDE. If you don't track the time spent paying down debt, the sheer volume of AI-generated code will eventually bankrupt your architecture. The Takeaway AI coding assistants are a massive net positive for the industry. But they are a hyper-efficient engine, and an engine is dangerously useless if you don't upgrade your brakes and steering at the same time. Stop looking at how fast your developers are closing tickets. Start looking at the health of your review culture, the speed of your pipelines, and the stability of your architecture. That is where the real engineering management happens today.

Last spring, I had six small text features to build: flag filler phrases in a draft, score sentence-length variation, format a citation, check a document against a rubric. My first design put all six behind an API route that called a model. It worked in an afternoon. Then I priced it. Anthropic lists Claude Fable 5 at $10 per million input tokens and $50 per million output. A 700-word draft plus instructions runs about 1,500 input tokens, and users hit the button five or six times per session while they edit. The bill is survivable. The rest of the tradeoff is not. Every keystroke a user typed would leave their machine and land in someone else's logs. Every click added 900ms of round trip to something that should feel like a spellchecker. And two runs over identical input returned different advice, which turns "did my edit help?" into an unanswerable question. I rewrote all six as deterministic browser code. No API route, no server, no network. This is what that took, and where the approach breaks. What a Heuristic Actually Catches The honest framing is that heuristics and models solve different problems, and half the features people route to an LLM belong in the first category. A model is worth paying for when the task needs world knowledge or judgment: Is this argument coherent, does this paragraph follow from the last one, is this claim supported? A regular expression cannot do any of that. But "does this text contain the phrase in order to" is a lookup. "How much do sentence lengths vary" is arithmetic. "Should of be capitalized in this title" is a rule from a style manual, written down, unchanged since 2019. Sending those to a probabilistic system buys you latency and nondeterminism in exchange for nothing. The six tools I run in production all fall in the second category. They ship as static pages with inline scripts, no build-time secrets, and no runtime dependencies. Sentence Segmentation Without a Regex You Will Regret Every metric below needs sentence boundaries, so this is the piece to get right first. Splitting on /[.!?]+\s+/ collapses under real prose. Run it over four ordinary lines and watch: Code language: Text Plain Text IN : The file cost $3.50. It shipped on Jan. 5 anyway. naive: ["The file cost $3.50", "It shipped on Jan", "5 anyway."] IN : He said "stop." Then he left. naive: ["He said \"stop.\" Then he left."] One false split, one missed split, and the abbreviation list you are about to write will never end. The browser ships an ICU-backed segmenter instead: Code language: JavaScript JavaScript const SEG = new Intl.Segmenter('en', { granularity: 'sentence' }); const raw = (text) => [...SEG.segment(text)].map((s) => s.segment.trim()).filter(Boolean); ICU gets both of those cases right, along with 9 a.m., decimals and section numbers like 2.1. It has one failure I hit in production, and it is worth knowing before you ship: it breaks after title abbreviations. Code language: Text Plain Text IN : She met Dr. Chen last week. The draft grew by 3.5 pages. ICU : ["She met Dr.", "Chen last week.", "The draft grew by 3.5 pages."] The repair is a merge pass over the output rather than a rewrite of the splitter. If a segment ends in a known title, glue the next one onto it: Code language: JavaScript JavaScript const TITLE_END = /(^|\s)(Dr|Mr|Mrs|Ms|Prof|Sr|Jr|St|vs|Fig|No)\.$/i; function sentences(text) { return raw(text).reduce((out, part) => { const prev = out[out.length - 1]; if (prev && TITLE_END.test(prev)) out[out.length - 1] = `${prev} ${part}`; else out.push(part); return out; }, []); } Verified against the cases above: Code language: Text Plain Text ["Dr. Chen wrote 3.5 pages.", "She revised twice."] ["She met Dr. Chen last week.", "The draft grew by 3.5 pages."] ["The file cost $3.50.", "It shipped on Jan. 5 anyway."] ["We deployed at 9 a.m.", "Nobody noticed."] ["He said \"stop.\"", "Then he left."] ["Prof. Ada Lovelace vs. Mr. Babbage.", "Round one."] That is a twelve-entry list against the open-ended one the naive regex demands, because ICU already covers the numeric and punctuation cases that make abbreviation lists grow. Intl.Segmenter landed in Chrome 87, Safari 14.1 and Firefox 125, so a 2026 audience has it. It also does granularity: 'word', which matters the moment a user writes in Thai or Japanese, where whitespace tokenization returns one enormous token. Guard it if you support older embedded webviews: Code language: JavaScript JavaScript const hasSegmenter = typeof Intl !== 'undefined' && 'Segmenter' in Intl; Phrase Matching That Does Not Fire on Substrings The naive filler checker uses indexOf, then reports "just" inside "adjustment" and loses the user's trust in the first thirty seconds. Build one alternation with word boundaries, compile it once, and keep the phrase list in data rather than code: Code language: JavaScript JavaScript const FILLERS = [ 'in order to', 'it is important to note', 'at the end of the day', 'due to the fact that', 'a wide variety of', 'needless to say', ]; const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); const FILLER_RE = new RegExp( '\\b(' + FILLERS.map(escapeRe).join('|') + ')\\b', 'gi' ); function findFillers(text) { return [...text.matchAll(FILLER_RE)].map((m) => ({ phrase: m[0], index: m.index, })); } Two details that cost me a rewrite. Compile the RegExp outside the function, because a global-flagged regex carries lastIndex state and rebuilding it per call hides that bug instead of fixing it. And use matchAll rather than a while (re.exec()) loop, which is where that state bites. The phrase list is the whole product here. Mine came from marking up 200 real drafts by hand, not from asking a model what filler looks like. Measuring Variation, and the Trap Next to It Uniform sentence length reads as flat prose. The metric is standard deviation over word counts: Code language: JavaScript JavaScript function rhythm(text) { const lens = sentences(text).map((s) => s.split(/\s+/).length); if (lens.length < 2) return null; const mean = lens.reduce((a, b) => a + b, 0) / lens.length; const variance = lens.reduce((a, n) => a + (n - mean) ** 2, 0) / lens.length; return { mean, sd: Math.sqrt(variance), count: lens.length }; } Low standard deviation is a useful writing signal. It is also, and this is where teams get into trouble, one of the two features commercial AI-text detectors lean on, alongside token-level perplexity. Do not ship it as one. A peer-reviewed study in Patterns tested seven commercial detectors and found they misclassified more than half of TOEFL essays written by non-native English speakers as machine-generated, while scoring near-perfect on native-speaker samples (full text). Steady sentence patterns are what a second-language writer produces under pressure. If your product tells that user their own writing looks synthetic, you have built a discrimination engine with a progress bar on it. Report the number as rhythm. Let the writer decide. Make "No Network" a Test, Not a Promise Claiming a tool runs locally is easy. Proving it survives the next dependency bump is the engineering. Two layers. Content Security Policy on the tool pages: Code language: HTML HTML <meta data-fr-http-equiv="Content-Security-Policy" content="default-src 'self'; connect-src 'none'; img-src 'self' data:;"> connect-src 'none' kills fetch, XMLHttpRequest, WebSocket and sendBeacon. If you run first-party analytics on the same origin, drop to connect-src 'self' and lean harder on the second layer. That second layer is a Playwright spec that fails the build if anything leaves the origin: Code language: JavaScript JavaScript test('clarity checker makes no offsite requests', async ({ page }) => { const offsite = []; page.on('request', (req) => { if (new URL(req.url()).origin !== BASE) offsite.push(req.url()); }); await page.goto(`${BASE}/tools/clarity-checker/`); await page.fill('#draft', 'In order to be clear, it is important to note this.'); await page.click('#analyze'); expect(offsite).toEqual([]); }); This caught a real regression for me: a font subset I added later pulled from a CDN, which meant the browser advertised the visitor's IP and user agent to a third party on a page whose whole selling point was that nothing left the device. The CSP would have blocked the request in a browser that enforced it. The test told me before a user did. The Comparison, With Numbers LLM API routeBrowser heuristicFirst response600–1,200 msunder 5 msMarginal cost~$0.001 per runzeroSame input, same outputnoyesUser text leaves deviceyesnoWorks offlinenoyesHandles novel phrasingyesnoJudges argument qualityyesnoShips without a backendnoyes The last row decided it for me. Six static pages on a CDN have no runtime to patch, no key to rotate, and no bill that scales with traffic. When to Call the Model Anyway I still reach for one, on three conditions. The task needs judgment rather than lookup. Restructuring an argument, catching a claim the writer never supported, spotting that paragraph four repeats paragraph two. No word list gets there. The user asked for it explicitly, with the data boundary stated in plain language on the button. Silent exfiltration dressed as a feature is how teams end up in a compliance review. And the output gets checked. For anything structured, constrain the response with a schema and validate it before it touches your UI, because a model that returns prose where your parser expects an object will do it on a Friday. Everything else stayed in the browser. Six features, roughly 400 lines of JavaScript total, zero infrastructure, and a p99 that is a rounding error. The default in 2026 is to reach for an API key first. Check whether the problem is a lookup before you do.

A chat screen looks like a weekend project: a list of bubbles and a text input pinned to the bottom. In React Native, it is one of the hardest things to ship well, because it sits on top of the two most hostile surfaces in mobile development: the software keyboard and a scrolling list that changes size while you're looking at it. We're putting LLMs into everything now, and there is still no good drop-in chat view for React Native. You glue together an aging library with strong opinions, or you hand-roll it. I hand-rolled it. Then I made the LLM stream its replies token by token, and the whole thing fell apart in a way that took a week to understand. This is the story of that break, and the fix, which arrived with suspicious good timing as a library release three months ago. The App I work on an app built around an LLM chat: characters that remember you and reply as an open-ended story unfolds. The messages between the reader and the characters are rendered in a chat-like view: an inverted list, the newest message at the bottom, and a composer pinned above the keyboard. Standard chat anatomy. The twist that makes it hard: the character replies are generated by an LLM, and they stream. Tokens arrive in bursts, a few every hundred milliseconds, with a full reply landing over two or three seconds. Each batch makes the last bubble taller. The list isn't just appending a finished message. It's growing on every frame, while the user might be typing, scrolling, or dismissing the keyboard. That single fact is what turns "I'll just use a FlatList" into weeks of work. Why There's Nothing Good to Reach For The first thing I did was look for a library. The honest state of the art: react-native-gifted-chat is the default answer and it's showing its age. It's opinionated about your data shape, its rendering, and its layout, and fighting those opinions costs more than writing your own.Most "chat UI" packages are really just a styled `FlatList` plus a text input. They solve the easy half and hand you the two genuinely hard problems: keyboard choreography and a live-resizing list.The keyboard utilities that _do_ exist (`KeyboardAvoidingView` and friends) were built for forms, not for an inverted list whose last row is growing while the keyboard animates. So I wrote my own keyboard-and-scroll layer. It was close to 500 lines of KeyboardAvoidingView overrides, manual scrollToOffset calls, listeners on keyboard show/hide events, and offset math to keep the composer glued to the keyboard. It worked, demos looked clean, and I shipped it. The Break: Streaming Meets the Keyboard The bug reports were all variations on "the chat is jumpy." No crashes, just jank. I couldn't reproduce it at first because each of the two features behaved perfectly on its own. The keyboard animation was smooth. The streaming was smooth. The problem only showed up at their intersection. That's the kind of bug that costs a week, because nothing is actually broken. Two correct things are simply disagreeing. Here's what was actually happening. While a character reply streams in: Every batch of tokens makes the last bubble taller.On an inverted list, growing the bottom row shifts the content offset.React Native re-runs the layout to absorb the new height.If the keyboard is open, or worse, mid-animation, my keyboard layer is _also_ adjusting offsets at the same time. Two systems are writing to the scroll position on the same frames. The result: the content jumps, the composer twitches, and if the user has scrolled up to re-read an earlier message, the stream yanks them around. Layout thrash. A steady 60fps collapsed into the low teens precisely when the app is supposed to feel most alive, and on a mid-range Android phone, it was worse. TypeScript // The naive streaming append: looks innocent, thrashes layout. // Every chunk triggers a re-measure of the growing bubble, // which fights whatever the keyboard handler is doing this frame. for await (const chunk of stream) { setMessages((prev) => { const next = [...prev]; next[0] = { ...next[0], text: next[0].text + chunk }; // index 0 = newest, inverted list return next; }); } The streaming itself has its own sharp edges, and they compound the layout problem. Two worth calling out before the fix: React Native's fetch can't stream a response body. There's no response.body.getReader() in stock RN. You reach for an SSE polyfill like react-native-sse or if you're on Expo like me, the streaming-capable fetch from expo/fetch. Pick deliberately. This is the single most common thing people get wrong on day one. TypeScript import { fetch } from "expo/fetch"; const res = await fetch(url, { method: "POST", body, signal: controller.signal, }); const reader = res.body.getReader(); const decoder = new TextDecoder(); // ...read loop, parse SSE frames, dispatch tokens Partial markdown will bite you. Tokens arrive mid-syntax. At some frame, your buffer is literally The dragon turned and **stared with the bold marker opened and not yet closed. A naive markdown renderer will either render the asterisks as literal text or flip half the conversation bold. You need a renderer that tolerates unterminated syntax, or you sanitize the buffer before each render. Cancellation has to be real. The user closes the chat, switches characters, or fires off a new message mid-reply. You need an AbortController whose signal actually reaches the fetch. Skip it and you're billed for tokens nobody will read, streamed into a view that already unmounted. The Fix I was about to rewrite my keyboard layer for the fourth time when react-native-keyboard-controller shipped KeyboardChatScrollView in v1.21.0, on March 16, 2026. It is, as far as I can tell, the first component built specifically for the chat-plus-keyboard problem rather than the form-plus-keyboard one, and it happens to solve the streaming case directly. The piece that matters for an LLM app is built on a ClippingScrollView that provides cross-platform contentInset behavior by extending the scrollable geometry rather than recomputing the layout. That one design choice is why the thrash disappears. The keyboard no longer fights the list because absorbing keyboard height is no longer a layout operation. The props read like a tour of every chat app you've used: keyboardLiftBehavior picks how the content reacts to the keyboard. "always" keeps the latest messages visible no matter where you've scrolled (Telegram, WhatsApp). "whenAtEnd" lifts only when you're already at the bottom, and leaves you alone if you've scrolled up to read history (ChatGPT). "persistent" lifts when the keyboard opens and, unlike the rest, stays put when it closes instead of snapping back down (Claude). "never" lets the keyboard cover the content and moves nothing (Perplexity).blankSpace reserves room for an incoming response while absorbing keyboard height. This is the direct antidote to streaming jank. Instead of the list growing reactively frame by frame and fighting the keyboard, you reserve the space up front and let the tokens fill it.extraContentPadding handles a composer that grows as the user types a long message, without jumping the content.freeze locks the layout during emoji and attachment-picker transitions, the other place chat UIs jump. TypeScript import { KeyboardChatScrollView } from "react-native-keyboard-controller"; <KeyboardChatScrollView keyboardLiftBehavior="persistent" // the Claude pattern: lifts on open, stays put on close blankSpace={pendingReply ? estimatedReplyHeight : 0} > {messages.map(renderBubble)} </KeyboardChatScrollView>; On paper whenAtEnd is the tidy answer for a reading-heavy app: don't move the content out from under someone studying an old exchange. I shipped persistent anyway. So many of my users live in assistant apps that Claude's settle-and-stay behavior is just what their hands expect, and familiarity beat theory. Nobody had to relearn how the chat feels. My streaming loop didn't change. What changed is that the loop is now the only thing touching layout while a reply comes in. The keyboard handler stepped out of the fight. The composer stopped twitching. The user who scrolls up to re-read an old exchange stays put while the character keeps talking below the fold. What I'd Keep, and What I'd Throw Away If I were starting Y/N's chat today, I'd delete my hand-rolled keyboard layer without ceremony and start from KeyboardChatScrollView. The custom code I'd keep is the part that was always mine to own: the streaming reader, the partial-markdown guard, and the cancellation plumbing. Those aren't keyboard problems, and no layout library will solve them for you. The general lesson applies well beyond chat. The expensive bug is almost never one broken feature. It's two correct features interacting on the same frame. My keyboard handler was right. My streaming was right. The week disappeared into the seam between them. When something janks and every part tests clean in isolation, stop testing the parts and go look at what they're both writing to. And the smaller, practical one: the chat box is never the easy part of the app. Budget for it like it's a feature, because it is one. For the first time in a while, you don't have to build all of it yourself. If you've solved the Android side of this, or made partial-markdown rendering feel good while streaming, I'd be glad to compare notes in the comments.

On one of our projects, we were building microfrontends, and at some point we wanted to add SSR. The reasons were the usual ones: better first paint, fewer layout shifts, real content for crawlers, less JS to load before something appears on screen. Setting it up turned out to be harder than I expected. There was no obvious out-of-box path that fit our setup, and most of the approaches I found either assumed a shared build or asked us to add new infrastructure on top of what we already had. That is what made me start sketching a small package. Something any team could drop in and get SSR for their microfrontend without rewriting either side. The result is @mf-toolkit/mf-ssr. The rest of this is about the approach behind it, since I think that is the interesting part. What I Wanted I started from a short list, taken straight from how I'd want to use such a thing: MF content on first paint. The remote's HTML should arrive inside the host's server response, not be fetched from the client after JS loads. No empty slot, no layout shift, real content in crawlers.No shared build, no central orchestrator. Each team builds and deploys their remote on their own schedule. The host should not need a special Node process that imports every remote into one bundle, and remote teams should not need to rewrite their bundler config to fit a central setup.Two paths for two setups, one host component. I wanted both scenarios covered. url mode for when the remote team runs their own server and wants to own SSR on their side (and possibly use a non-React framework). loader mode for when the remote only ships a static React bundle and the host server can do the SSR for it. The host code should look almost the same in either case, with just a single prop telling the component which path to use.Any framework, any runtime. The remote might be React, but it could be Vue, Svelte, or anything else. The host shouldn't care. And on the server, the same code should run on Node, Bun, Cloudflare Workers, or Vercel Edge with no rewrites.Host state still drives the remote after hydration. When the host re-renders with new props, the remote should re-render too. No re-fetch, no re-mount, no shared store between bundles.Honest failure modes. A timeout when the remote is slow, retry when a request fails, an explicit fallback for total failure, and a cache that respects auth boundaries. The things that decide whether SSR is a win or a regression when one team has a bad deploy. The last bullet is what most articles skip. SSR is easy in the happy path. The interesting code is what happens when one of the remotes is slow, down, or returning garbage. How It Works The idea is small: Instead of importing remote components into the host server, the host pulls the rendered output in over HTTP at SSR time and streams it into its own response. The browser gets a full page on first paint. How that "pull" happens depends on how the remote is deployed. The package supports two modes for that: url mode – the remote has its own HTTP endpoint that returns rendered HTML. The host fetches that HTML during SSR.loader mode – the remote is a static React bundle on a CDN or S3, no server behind it. The host imports the component directly during SSR and renders it inline. Same host component (<MFBridgeSSR>) in both cases, just one prop changes. Both modes can live on the same page. The interesting part is what happens after hydration. The host has to push prop changes into the remote without re-fetching anything. I will get to that in a moment. I'll start with url mode since it is the more general case (any framework on the remote side, any runtime on the server), and then cover loader mode separately. url mode: Remote With Its Own HTTP Endpoint In url mode, the remote server does the SSR. The remote team runs their own runtime (Node, Bun, a Cloudflare Worker, a Next.js Route Handler, whatever they prefer) and exposes an HTTP endpoint that returns rendered HTML for the given props. The host's SSR pass just calls that endpoint and inlines the response into the page. Each microfrontend owns its own rendering pipeline. Remote Handler TypeScript-JSX import { createMFReactFragment } from '@mf-toolkit/mf-ssr/fragment' import { CheckoutWidget } from './CheckoutWidget' export const handler = createMFReactFragment(CheckoutWidget) handler is a plain Web fetch handler: (req: Request) => Promise<Response>. It reads props from the query string, renders the component to a stream with renderToReadableStream, and writes the props into a small <script> tag so the client can hydrate without going back to the network. One nuance worth flagging: those props go inside a <script> tag, so a raw </script> inside a string prop would close the tag prematurely and let user-controlled values escape into the HTML context. The handler escapes <, >, &, and U+2028/U+2029 to their \uXXXX equivalents before embedding. JSON.parse on the client treats them the same as the originals, but the browser's HTML parser never sees a closing tag. It is a few lines of code that close a real XSS hole. You wire the handler into whatever HTTP framework the remote team already uses. Hono, a Next.js Route Handler, Bun, plain Node, a Cloudflare Worker. The handler doesn't know about any of them. And because the whole thing is Web Streams, it runs on Cloudflare Workers, Vercel Edge, Bun, and Node 18+ without changes. Non-React Remotes createMFReactFragment is a React-only helper. If the remote is Vue, Svelte, Solid, or vanilla JS, the team writes their own fetch handler instead, but it has to produce the same HTML shape the host expects: TypeScript-JSX <div data-mf-ssr="checkout"> <script type="application/json" data-mf-props>{"orderId":"42"}</script> <div data-mf-app><!-- Vue / Svelte / whatever rendered HTML --></div> </div> The team uses their framework's SSR renderer (renderToString for Vue, Svelte's SSR API, and so on) to produce the inner HTML, and serializes props into the <script data-mf-props> tag, applying the same < / > / & escaping. On the client, the remote mounts itself into [data-mf-app] and reads initial props from [data-mf-props]. If it needs prop updates from the host after hydration, it listens on the same DOMEventBus (exported from @mf-toolkit/mf-bridge). The bus is a thin wrapper over native CustomEvent, with no React dependency, so it works fine for any framework. This path is more work than createMFReactFragment, but the contract is small and explicit. The host doesn't care which framework produced the inner HTML — as long as the wrapper structure matches, hydration finds the right slots. Host Component TypeScript-JSX <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId, step } fallback={<CheckoutSkeleton />} /> During SSR, the host fetches the remote's HTML and streams it into the response. Each <MFBridgeSSR> lives in its own Suspense boundary, so a slow checkout doesn't block the header. They stream as they resolve. On the client, the host hydrates, then waits for prop changes coming from React. Prop Updates After Hydration This was the part I cared about most. The remote is in its own React root, often in its own bundle, sometimes in a completely different framework. You can't re-render it like a normal child. So I used the one thing both sides already share at runtime: the DOM node the remote is mounted into. When the host re-renders with new props, the host fires a CustomEvent on that node. The remote listens for it and re-renders its root with the new props. No re-fetch, no global state, no coupling between bundles beyond a shared namespace string. TypeScript-JSX // remote client entry import { hydrateWithBridge } from '@mf-toolkit/mf-bridge/hydrate' import { CheckoutWidget } from './CheckoutWidget' hydrateWithBridge(CheckoutWidget, { namespace: 'checkout' }) I picked this because it is isolated by construction. If a page has several MF slots, each one has its own mount node, so events never leak between them. And it is just DOM, so there is no bundler magic to debug when something goes wrong. Events and Commands Prop streaming is one direction. For the other direction, the same bus works in reverse. The host passes onEvent to receive events the remote emits, and a commandRef it can use to send imperative commands back: TypeScript-JSX const resetRef = useRef<((type: string, payload?: unknown) => void) | null>(null) <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId } onEvent={(type, payload) => { if (type === 'orderPlaced') navigate('/thanks') } commandRef={resetRef} /> // somewhere in host code, e.g. when the user switches accounts: resetRef.current?.('reset') On the remote, hydrateWithBridge accepts an onCommand handler, and DOMEventBus (exported from @mf-toolkit/mf-bridge) lets the remote send events back: TypeScript-JSX import { hydrateWithBridge } from '@mf-toolkit/mf-bridge/hydrate' import { DOMEventBus } from '@mf-toolkit/mf-bridge' hydrateWithBridge(CheckoutWidget, { namespace: 'checkout', onCommand: (type) => { if (type === 'reset') store.reset() }, }) // inside the widget, after a successful payment: const container = document.querySelector<HTMLElement>('[data-mf-namespace="checkout"]')! new DOMEventBus(container, 'checkout').send('event', { type: 'orderPlaced', payload: { orderId }, }) The channel is the same DOMEventBus, just with extra event names on top of propsChanged. So everything I said earlier about isolation still holds: events on one slot don't reach another, even when the remote is the same. loader mode: Remote as a Static Bundle In loader mode, the host server does the SSR for the remote. The remote team ships only a static React bundle (CDN, S3, or a Module Federation host) and runs no server of their own. When the host renders its page server-side, it imports the remote component and renders it inline, the same way it renders any other component in the host tree. The remote has no SSR runtime and no rendering responsibility; the host does all the work.ё Host Component JSX const loadCheckout = () => import('checkout/Widget').then(m => m.CheckoutWidget) <MFBridgeSSR loader={loadCheckout} props={{ orderId, step } fallback={<CheckoutSkeleton />} /> That is everything. No namespace, no errorFallback tricks needed for hydration, no client entry to write on the remote side. The package wraps the loader in React.lazy and renders the component inside the host's React tree, both server-side and after hydration. Props, Events, Commands Since the remote lives inside the host's React tree, every kind of communication is just React: Props – re-render normally. When the host's parent component re-renders with new props, the remote re-renders too. No DOMEventBus, no hydrateWithBridge, no propsChanged events.Events from remote to host – pass a callback through props. The remote calls it like any other handler.Commands from host to remote – pass them through props as well, or expose a ref through forwardRef. If you find yourself wanting onEvent / commandRef here, you are probably reaching for url mode. Requirements A few constraints come with this mode: Host must be able to resolve the loader on the server. The package calls your loader() function as-is. It doesn't fetch bundles from URLs itself. In practice, this means Module Federation runtime on the host (or some other server-side dynamic import mechanism that knows how to find checkout/Widget). Without that, the import fails in Node before any rendering happens.React only. The host literally calls the component during SSR, so the remote has to be a React component. For Vue/Svelte/vanilla remotes, use url mode.SSR-safe import. The remote's exposed module has to be importable on the server, which means no window, document, or other browser globals at the module top level. Move that code inside useEffect or behind a typeof window check.Stable loader reference. Define loadCheckout at module scope or wrap it in useCallback. The package caches the resulting React.lazy by loader reference so Suspense retries reuse the same promise. A new function on every render would break that and trigger an infinite retry loop. When to Pick Which CategoryURL modeLoader modeRemote infrastructureOwn HTTP endpoint: Node.js, Bun, Worker, etc.Static bundle on CDN, S3, or Module Federation hostRemote frameworkAny: React, Vue, Svelte, vanilla JavaScriptReact onlyIsolationSeparate React root inside the remote bundleRendered inline in the host React treeProp updatesDOM events through DOMEventBusNative React re-renderEvents and commandsonEvent and commandRefReact props and refsBest forIndependent teams, mixed frameworks, and polyreposSimple React remotes with no extra infrastructure Both modes use the same <MFBridgeSSR> and can be mixed freely on the same page. The Corner Cases I Spent Time On A few production scenarios I wanted to make sure the package handled honestly. Graceful Degradation When the Remote Is Down A remote can be slow, return a 5xx, or simply not respond. The host page shouldn't break because of one bad slot. mf-ssr accepts an errorFallback, and the trick is that the fallback can be the same remote mounted on the client through mf-bridge: TypeScript-JSX import { MFBridgeSSR } from '@mf-toolkit/mf-ssr' import { MFBridgeLazy } from '@mf-toolkit/mf-bridge' <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId } timeout={2000} errorFallback={ <MFBridgeLazy register={() => import('checkout/entry').then(m => m.register)} props={{ orderId } fallback={<CheckoutSkeleton />} /> } /> If the SSR fetch times out, the user still gets the widget. Just on the client, the same way it would have worked without mf-ssr at all. The page doesn't break. The slot loses its first-paint optimization, for that one request. When the remote recovers, the next render uses SSR again with no code change on either side. I like this case because it inverts the usual SSR-or-nothing tradeoff. SSR becomes the fast path, with a working client-side path sitting right behind it. Auth-Isolated Caching The host caches fragments by url + props + timeout. Fine for public content. Not fine when each user gets different HTML — they would share a cache slot and see each other's pages. So there is a cacheKey prop you set when the request carries auth: TypeScript-JSX <MFBridgeSSR url="https://account.acme.com/fragment" namespace="account" props={{ view: 'orders' } fetchOptions={{ headers: { authorization: `Bearer ${token}` } } cacheKey={userId} /> The other side of the same coin is public fragments. The remote's fragment endpoint accepts a cacheControl option, so you can serve a product card as public, s-maxage=60, stale-while-revalidate=30 and let a CDN cache it for everyone: TypeScript-JSX export const handler = createMFReactFragment(ProductCard, { cacheControl: 'public, s-maxage=60, stale-while-revalidate=30', vary: 'Accept-Language', }) One pattern handles per-user fragments, the other handles cacheable public ones. Same component on both sides. Multiple Instances of the Same Remote Header, sidebar, and a content slot can all be the same remote on one page. The reason I sent prop updates through the mount DOM node, instead of a global event bus, is exactly this case: each <MFBridgeSSR> has its own DOM node, so events stay scoped to it. No filtering by instance id, no manual subscription bookkeeping. Warming the Cache From RSC If you know a fragment is going to be needed, you can start the fetch before <MFBridgeSSR> even renders. Suspense then skips the fallback entirely: TypeScript-JSX import { preloadFragment } from '@mf-toolkit/mf-ssr' // In a Server Component or route loader preloadFragment('https://checkout.acme.com/fragment', { orderId }) By the time the component renders down the tree, the HTML is already there. Where It Fits If your microfrontends share one build (a single bundler config that imports every remote), you don't need any of this. Use whatever your framework gives you. mf-ssr is for the case where each team builds and deploys independently. Different repos or not, the point is that there is no shared build step pulling everything into one Node process — and you still want a full page on first paint. The bet is that HTTP is a good enough boundary between teams, and that DOM events are a good enough way to keep host state in sync with remote rendering after hydration. The CSS isolation question, by the way, lives in mf-bridge, not here: it has shadowDom and adoptHostStyles props that wrap the remote in a Shadow DOM and forward host stylesheets (including Tailwind / CSS-in-JS chunks injected after mount) into the shadow root. SSR fragments don't use it by default since the HTML is inlined into the host response, but the option exists if you want it. Try It The package is published as @mf-toolkit/mf-ssr. The repo has runnable examples, and I've also made a demo repo where you can play with all my tools. If you've solved the same problem in a different way, I'd be curious to compare notes.

The Problem: Our p99 Was 3-5 Seconds Our PyFlink pipeline was missing its latency SLO by seconds. The pipeline itself was straightforward: consume events from Kafka, transform them, serialize them as Protobuf, and write the results to downstream systems. Yet under production load, p99 end-to-end latency was consistently in the 3-5 second range. Profiling pointed us to an unexpected bottleneck: we were deserializing Protobuf messages in Python, even though the Flink runtime processing our stream was JVM-based. Every record that entered the Python path had to cross the JVM-to-Python process boundary, get parsed by a Python UDF, and then cross back. The business logic wasn't the problem. The doorway was. We moved Protobuf deserialization to Flink's JVM-side Protobuf format and kept Python for orchestration and SQL. In our environment, p99 dropped to approximately 500 milliseconds, with less code and a pipeline that is easier to reason about. Verified on AWS Managed Service for Apache Flink (formerly Kinesis Data Analytics). Why Python-Side Deserialization Is So Expensive The naive PyFlink architecture looks like this: A Kafka source table declared with a generic format (raw, json, or a SimpleStringSchema), so every record arrives as opaque bytes or a string.A Python map() or UDF that imports generated _pb2.py classes and calls ParseFromString() on every message.Downstream transforms and sinks. Two costs hide in step 2, and they compound at high throughput. The process boundary. PyFlink is not Python running inside Flink; it is a JVM runtime coordinating with a separate Python execution environment. Every record that enters the Python execution path incurs overhead associated with moving data between the JVM and Python, and depending on the operator and execution mode, that can involve serialization and inter-process communication in both directions. For a per-record deserialization UDF on a latency-sensitive pipeline, that overhead is paid before the actual business transformation begins. Per-record parse cost. Even when Python's Protobuf implementation uses its native backend, parsing in a Python UDF still requires the record to enter the Python execution path. When the workload is latency-sensitive and high-throughput, the combination of serialization, inter-process communication, Python execution, and parsing overhead can become significant. In our case, profiling showed that this path was a major contributor to our latency. In our pipeline, these two costs together accounted for the bulk of the gap between a 3–5 second p99 and the ~500ms target we needed, before the enrichment logic even began executing. The Key Realization: PyFlink Already Runs on the JVM Here's the insight that changes the architecture: if Protobuf is declared at the table DDL level, Flink's Kafka connector deserializes it with its native, optimized JVM-based Protobuf format before any data reaches the Python side. Your columns simply arrive typed and ready. Python's role shrinks to what it's genuinely good at in this stack: orchestration and SQL. No rewrite to Java. No change to how jobs are deployed. Just a different declaration of intent. The trade is that Flink's native Protobuf format needs compiled Java message classes on the classpath; it does not consume .proto files or Python _pb2 modules directly. That means adding a small build step to your workflow, which we'll cover below. Implementation The pipeline splits into two declarative jobs. Job 1: JSON In, Protobuf Out The source table reads the raw JSON topic; the sink table declares format = 'protobuf' and points at the compiled Java class. The JVM handles typed-row-to-Protobuf encoding. SQL -- SOURCE: raw JSON payload as STRING plus Kafka record timestamp CREATE TABLE source_events_json ( event_data STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = '${INPUT_JSON_TOPIC}', 'properties.bootstrap.servers' = '${KAFKA_BOOTSTRAP_SERVERS}', 'scan.startup.mode' = 'latest-offset', 'format' = 'raw' ); -- SINK: Protobuf out to Kafka (JVM handles typed row to Protobuf) CREATE TABLE sink_events_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent' ); -- TRANSFORM: pure SQL, no Python UDFs INSERT INTO sink_events_pb SELECT JSON_VALUE(event_data, '$.id') AS id, JSON_VALUE(event_data, '$.organization_id') AS organization_id, ROW( UNIX_TIMESTAMP(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '')), CAST(EXTRACT(NANOSECOND FROM CAST(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '') AS TIMESTAMP_LTZ(9))) AS INT) ) AS event_ts, CAST(JSON_VALUE(event_data, '$.is_active') AS BOOLEAN) AS is_active, JSON_VALUE(event_data, '$.event_type') AS event_type FROM source_events_json; Note what's absent: no ParseFromString(), no _pb2.py imports, no Python deserialization loop. The Python program registers DDL and runs SQL. Job 2: Protobuf In, OpenSearch Out Downstream, the sanitized Protobuf topic becomes a typed source, using the same protobuf.message-class-name property, plus ignore-parse-errors so a malformed record can't poison the pipeline. SQL -- SOURCE: Protobuf from the sanitized Kafka topic CREATE TABLE kafka_source_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'scan.startup.mode' = 'latest-offset', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent', 'protobuf.ignore-parse-errors' = 'true' ); -- SINK: OpenSearch (JSON) CREATE TABLE opensearch_sink ( id STRING, organization_id STRING, event_ts TIMESTAMP_LTZ(3), is_active BOOLEAN, event_type STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'opensearch-2', 'hosts' = '${OPENSEARCH_ENDPOINT}:443', 'index' = 'acme-events-v1', 'format' = 'json' ); INSERT INTO opensearch_sink SELECT id, organization_id, TO_TIMESTAMP_LTZ(event_ts.seconds * 1000, 3), is_active, event_type FROM kafka_source_pb; The Build Step: Getting Java Classes Onto Flink's Classpath The one genuinely new piece of workflow is compiling your .proto definitions to Java and packaging them into the job's fat JAR. The essential Maven pieces: XML <dependencies> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.25.5</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-protobuf</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>${flink.connector.kafka.version}</version> </dependency> <!-- plus your sink connectors, e.g. flink-connector-opensearch2 --> </dependencies> Two practices that made this maintainable for us: Version-control the generated Java sources (or generate them in CI from a single canonical .proto repo) and pull them in with build-helper-maven-plugin's add-source, rather than compiling .proto files in every consuming project. One schema source of truth, many consumers.Shade everything into one JAR with maven-shade-plugin, excluding signature files (META-INF/*.SF, *.DSA, *.RSA). On AWS Managed Flink, pass it via the job's JAR configuration; on self-managed Flink, drop it in lib/ or use --classpath. The full workflow: define the .proto, compile it to Java with protoc, package the fat JAR, put it on Flink's classpath, author the PyFlink job with the DDL above, then deploy and watch end-to-end p99. How We Measured the Improvement We measured end-to-end p99 latency as the time from a record landing on the source Kafka topic to the corresponding OpenSearch write being acknowledged MetricBeforeAfterp99 latency3-5s~500msSustained throughput~5,000 events/sec~5,000 events/secFlink parallelism128Python UDF parsingYesNoJVM/Python boundary on hot pathYesNoProtobuf decodingPythonJVM Results End-to-end p99 latency around 500 milliseconds in our environment at production load, down from a 3-5 second baseline, by eliminating per-record JVM-to-Python crossings and Python-side parsing on the hot pathLess code. The deserialization UDFs, the _pb2 imports, and their error handling all disappeared. What remains is DDL plus SQLSimpler and easier to operate. The pipeline now relies on Flink's Kafka connector and Protobuf format for serialization and parsing, with built-in parse-error handling, instead of hand-rolled Python parsing When This Optimization Won't Help Moving Protobuf decoding to the JVM won't automatically solve every latency problem. If your pipeline's critical path is dominated by sink backpressure, network latency, external API calls, state access, or checkpointing overhead rather than deserialization, changing the serialization path may have little effect on end-to-end latency. This optimization is most valuable when profiling specifically shows that Python execution and JVM/Python data movement are significant contributors to the critical path, which is why we'd recommend profiling first rather than applying this as a default change. When You Should Still Use Python UDFs This pattern is not "never write Python UDFs." It's "keep them off the per-record deserialization path." Python remains the right tool when: The transformation genuinely needs Python libraries (ML feature computation, model inference, specialized parsing that has no SQL equivalent).Throughput is modest and developer velocity matters more than the last hundred milliseconds.You're prototyping. Even then, declare the format natively from day one anyway; it costs nothing and you won't have to migrate later. If a UDF is unavoidable on a hot path, at least let the JVM do the deserialization first so the UDF receives typed columns rather than raw bytes. Gotchas Worth Knowing Before You Ship Property syntax varies by Flink version. Some versions use format = 'protobuf'; newer key/value descriptors prefer value.format = 'protobuf'. Check your version's docs.Enums: surface them as STRING if you need ergonomic SQL manipulation, or keep them numeric with a lookup table.Schema evolution: favor backward-compatible, additive changes with defaults. Because the compiled Java classes are baked into the JAR, a schema change means a rebuild and redeploy, so make that a deliberate, versioned step in CI rather than an afterthought. ignore-parse-errors is your safety net during rollout windows, but monitor the drop counter so it doesn't silently eat data.Benchmark end-to-end, not just the UDF: source lag, operator latency, and sink acknowledgments under production load patterns. Deserialization wins can be masked, or dwarfed, by sink backpressure.Security: lock down OpenSearch credentials and TLS; pin Kafka client versions compatible with your Flink release. Closing Thoughts We didn't rewrite the pipeline in Java. We removed an unnecessary per-record JVM-to-Python boundary from the hot path and let Flink's JVM-native Protobuf format do the work it was designed to do. If your PyFlink job parses Protobuf messages in Python today, check whether Flink's native format support can move that work into the JVM-side execution path. For latency-sensitive pipelines, eliminating unnecessary Python boundaries may be one of the highest-leverage optimizations to investigate, especially when profiling shows that serialization and Python execution are on the critical path.

JWT-based authentication is simple to start with and surprisingly hard to get right. The naive setup of a long-lived access token stored in the browser is a security liability. The textbook fixes short-lived access tokens plus a refresh token — introduce their own problem: What Happens When a Refresh Token Is Stolen? This article walks through implementing refresh token rotation with reuse detection, a pattern that limits the damage of a stolen token while keeping legitimate users logged in. All examples are in Node.js with Express. Why a Single Long-Lived Token Is Dangerous If you issue one access token that lives for days, you have no way to revoke it before it expires. If it leaks through an XSS bug, a logging mistake, or a compromised device, an attacker has full access until expiry, and you cannot do anything about it. Short-lived access tokens (say, 15 minutes) limit this window. But you cannot ask users to log in every 15 minutes, so you pair the access tokens with a longer-lived refresh token whose only job is to mint new access tokens. The New Problem: Stolen Refresh Tokens A refresh token is now the crown jewel. If an attacker steals it, they can mint access tokens indefinitely. Simply making it long-lived recreates the original problem at a higher level. Rotation addresses this: every time a refresh token is used, it is invalidated, and a brand-new refresh token is issued. A stolen token is only useful until the legitimate user next refreshes, at which point the stolen token becomes invalid. But rotation alone is not enough. Consider the race: Attacker steals refresh token R1.Legitimate user refreshes with R1, gets R2. R1 is now invalid.Attacker tries R1. It is rejected, but the system does not yet know a theft occurred. The missing piece is reuse detection: if an already-rotated token is presented again, that is a strong signal of theft, and the entire token family should be revoked. Implementing Token Families The key concept is the token family. When a user logs in, you create a family with a shared family_id. Every rotation issues a new token in the same family. If any consumed token in the family is ever presented again, you revoke the whole family, forcing the attacker (and the victim) to re-authenticate. Here is the schema: SQL CREATE TABLE refresh_tokens ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), family_id UUID NOT NULL, user_id INTEGER NOT NULL, token_hash VARCHAR(255) NOT NULL, consumed BOOLEAN NOT NULL DEFAULT false, expires_at TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_family ON refresh_tokens (family_id); Note that we store a hash of the token, never the token itself. If your database leaks, the stored hashes are useless to an attacker, the same reasoning behind hashing passwords. Issuing Tokens at Login TypeScript const crypto = require('crypto'); const jwt = require('jsonwebtoken'); function hashToken(token) { return crypto.createHash('sha256').update(token).digest('hex'); } async function issueTokens(userId, familyId = crypto.randomUUID()) { const accessToken = jwt.sign( { sub: userId }, process.env.ACCESS_SECRET, { expiresIn: '15m' } ); const refreshToken = crypto.randomBytes(40).toString('hex'); const expiresAt = new Date(Date.now() + 7 * 24 * 60 * 60 * 1000); // 7 days await pool.query( `INSERT INTO refresh_tokens (family_id, user_id, token_hash, expires_at) VALUES ($1, $2, $3, $4)`, [familyId, userId, hashToken(refreshToken), expiresAt] ); return { accessToken, refreshToken, familyId }; } The Rotation Endpoint With Reuse Detection This is where the security logic lives: TypeScript async function rotateRefreshToken(presentedToken) { const hash = hashToken(presentedToken); const result = await pool.query( `SELECT * FROM refresh_tokens WHERE token_hash = $1`, [hash] ); if (result.rowCount === 0) { throw new Error('INVALID_TOKEN'); } const token = result.rows[0]; // REUSE DETECTION: a consumed token is being presented again. // This means the token was either stolen or replayed. Burn the family. if (token.consumed) { await pool.query( `DELETE FROM refresh_tokens WHERE family_id = $1`, [token.family_id] ); throw new Error('TOKEN_REUSE_DETECTED'); } if (new Date(token.expires_at) < new Date()) { throw new Error('EXPIRED_TOKEN'); } // Mark this token consumed, then issue a fresh one in the same family. await pool.query( `UPDATE refresh_tokens SET consumed = true WHERE id = $1`, [token.id] ); return issueTokens(token.user_id, token.family_id); } The crucial branch is the token consumed check. Under normal operation, a token is used exactly once and then never seen again. If a consumed token reappears, the only explanations are theft or a replay attack, so the system revokes every token in the family. The attacker is locked out, and the legitimate user is forced to log in again, which is the correct, safe outcome. A Common Mistake: Forgetting the Grace Window There is a subtle real-world wrinkle. Mobile clients on flaky networks sometimes fire two refresh requests for the same token because the first response was lost in transit. With strict reuse detection, the second request looks like an attack and nukes the family, logging out a user who did nothing wrong. The pragmatic fix is a short grace window: if a consumed token is reused within a few seconds of being consumed, return the already-issued replacement token instead of revoking the family. This tolerates network retries without weakening protection against real theft, which happens on a much longer timescale. TypeScript const GRACE_MS = 10_000; if (token.consumed) { const age = Date.now() - new Date(token.consumed_at).getTime(); if (age < GRACE_MS) { // Likely a network retry — return the existing replacement. return getReplacementToken(token.family_id); } // Otherwise, treat as theft. await revokeFamily(token.family_id); throw new Error('TOKEN_REUSE_DETECTED'); } (This requires adding a consumed_at timestamp and a pointer to the replacement token, omitted here for brevity.) Takeaways Keep access tokens short-lived (15 minutes is reasonable) and never rely on them being revocable. Rotate refresh tokens on every use so a stolen token has a short useful life.Detect reuse of consumed tokens and revoke the entire token family — this is what actually catches theft.Store only hashes of refresh tokens, never the raw values.Add a small grace window so flaky-network retries do not get misread as attacks. Rotation with reuse detection is more code than a plain JWT setup, but it turns authentication from "hope nothing leaks" into a system that actively detects and contains compromise.

I maintain a React admin dashboard codebase that had — at last count before upgrading to React 19 — 34 instances of useMemo, 28 instances of useCallback, and 19 components wrapped in memo(). I spent a nontrivial amount of time over two years adding those optimizations, debugging cases where I'd gotten the dependency arrays wrong, and explaining to junior developers why the table re-rendered on every keystroke. React 19 with the compiler deleted most of that work. Here's what actually changed and what still matters. The Compiler Handles What You Used to Do Manually The React Compiler analyzes component render behavior and adds memoization where it's beneficial automatically. You don't specify it. You don't maintain dependency arrays. You don't wrap components in memo(). Before: JSX const Dashboard = memo(function Dashboard({ userId, filters }) { const processedData = useMemo( () => processData(rawData, filters), [rawData, filters] ); const handleFilterChange = useCallback( (newFilter) => updateFilters(newFilter), [updateFilters] ); return <DataTable data={processedData} onFilter={handleFilterChange} />; }); After React 19 with compiler: JSX function Dashboard({ userId, filters }) { const processedData = processData(rawData, filters); const handleFilterChange = (newFilter) => updateFilters(newFilter); return <DataTable data={processedData} onFilter={handleFilterChange} />; } Same performance. Half the code. Zero dependency array bugs. I removed 31 of my 34 useMemo calls after upgrading. The three I kept are genuinely complex computations where I want explicit control. Everything else the compiler handles better than I was doing manually. The One Thing That Still Kills Dashboard Performance The compiler doesn't solve virtualization. If your data table renders 5,000 DOM nodes because your dataset has 5,000 rows — React 19 won't fix that. The browser is still creating and painting 5,000 nodes. Any table with more than 100 rows needs virtualization: JSX import { useVirtualizer } from '@tanstack/react-virtual'; export default function DataTable({ rows }) { const parentRef = useRef(null); const virtualizer = useVirtualizer({ count: rows.length, getScrollElement: () => parentRef.current, estimateSize: () => 52, overscan: 5 }); return ( <div ref={parentRef} style={{ height: '600px', overflow: 'auto' }> <div style={{ height: virtualizer.getTotalSize() }> {virtualizer.getVirtualItems().map(row => ( <div key={row.index} style={{ position: 'absolute', top: 0, transform: `translateY(${row.start}px)`, height: row.size } > <TableRow data={rows[row.index]} /> </div> ))} </div> </div> ); } 5,000 rows. 20 DOM nodes in the viewport at any time. Scroll is smooth. The React Compiler cannot help you here — this is a DOM problem, not a React problem. Optimistic Updates Changed How My Dashboard Feels The biggest perceived performance improvement in React 19 for admin dashboards isn't the compiler. It's useOptimistic. Admin dashboards involve constant small mutations — toggling user status, updating values, changing settings. Before React 19, every mutation waited for the server response before updating the UI. Fast servers meant 200ms delays. Slow servers meant users clicking buttons twice because nothing happened visually. JSX 'use client'; import { useOptimistic, useTransition } from 'react'; function UserStatusBadge({ user, onUpdateStatus }) { const [optimisticStatus, setOptimisticStatus] = useOptimistic( user.status, (_, newStatus) => newStatus ); const [isPending, startTransition] = useTransition(); const toggle = () => { const newStatus = optimisticStatus === 'active' ? 'inactive' : 'active'; startTransition(async () => { setOptimisticStatus(newStatus); // instant UI update await onUpdateStatus(user.id, newStatus); // server in background }); }; return ( <button onClick={toggle} disabled={isPending} className={`badge ${optimisticStatus === 'active' ? 'bg-success' : 'bg-secondary'}`} > {optimisticStatus} </button> ); } Click. Status changes instantly. The server call happens in the background. If the server fails — React reverts the optimistic update automatically. The dashboard feels instant because it is instant from the user's perspective. I added this to every status toggle, every inline edit, every bulk action in my dashboard. The difference in perceived performance is more noticeable to users than any memoization optimization I'd done previously. What React 19 Didn't Fix Route-level code splitting still matters and still requires explicit configuration. If your dashboard loads all route components upfront, the initial bundle is large regardless of React version: JavaScript // Still need this in React 19 const UsersPage = lazy(() => import('./pages/Users')); const AnalyticsPage = lazy(() => import('./pages/Analytics')); const SettingsPage = lazy(() => import('./pages/Settings')); Image optimization still requires next/image or equivalent. Unoptimized images are still the most common performance problem I see in dashboard codebases, and React 19 does nothing about them. Database query performance still determines how fast your data loads. The fastest React rendering in the world doesn't compensate for a 3-second API response. The Summary React 19 removes the memoization overhead that made large React codebases tedious. Use the compiler, stop writing useMemo for everything, and trust that it handles the cases you were handling manually. What still requires your attention: virtualize large tables, add optimistic updates for frequent mutations, split routes with lazy loading, and optimize your images. The performance work that remains is more interesting than the work React 19 eliminated. That's a good trade.

In modern application development, feature flags are the guardrails that keep experiments controlled and rollbacks safe when conditions shift. If feature flags act as the guardrails, observability provides the visibility: the headlights (traces), mirrors (logs), and dashboard instruments (metrics) that reveal what’s happening in the environment and how well a feature is performing. Together, feature flags and observability unlock powerful insights by correlating code changes with real-time system behavior. This combination reduces time-to-diagnosis and builds greater confidence when rolling out new features. In this post, we’ll walk through just how to add observability to a React Native application using LaunchDarkly’s observability SDK. To demonstrate the process, we’ll build on the PlusOne app, a simple counter app that includes increment (+1), reset, and error-triggering buttons. This lightweight demo provides a clean foundation to showcase how logs, traces, and errors can seamlessly flow into LaunchDarkly for monitoring and debugging. Prerequisites LaunchDarkly account. Sign up for a free one here.Visual Studio or another code editor of choice. All code from this tutorial can be found on GitHub. Setting Up Your Environment Before running a React Native app, make sure your development environment is set up correctly. You can find the full setup instructions for both Android and iOS here. In this tutorial, we'll be running iOS, but keep in mind Expo Orbit, the platform we'll be using to run our iOS simulator, requires both Xcode and Android Studio to be installed. After going through the instructions, you should have the following installed: Node JS (preferably via nvm)Watchman for file monitoringJDK via zulu package managerAndroid Studio. Don’t forget to set your Android_Home environment variablesXcode for the iOS simulatorCocoapods for iOS dependency managementExpo Orbit for running Expo apps on Android or iOS If you're using Android, don't forget to add your environment variables to bash or zsh profile. JavaScript export ANDROID_HOME=$HOME/Library/Android/sdk export PATH=$PATH:$ANDROID_HOME/emulator export PATH=$PATH:$ANDROID_HOME/platform-tools Starting Up the PlusOne App To get started, let’s clone the repo for the PlusOne app and run npm install to ensure the proper dependencies are present in our node_modules file. Clone the repo. JavaScript git clone https://github.com/arober39/PlusOne Install dependencies using npm. JavaScript cd PlusOne npm install We’ll also need to run both the prebuild command to generate the iOS file and the expo run command to run the iOS simulator. Prebuild for iOS. JavaScript npx expo prebuild Run expo app. JavaScript npm expo run:ios Now we can view the iOS app in the iPhone simulator using npm. JavaScript # iOS npm run ios # Android npm run android The app should look something like this: Feel free to interact with the app to ensure all is working as expected. As you can see in the code, we have three buttons: one that adds one to the displayed number, one to bring the count back to zero, and an intentional Error button to test error monitoring within the LaunchDarkly UI. JavaScript // app/index.tsx import { useState } from "react"; import { StyleSheet, Text, TouchableOpacity, View } from "react-native"; export default function Index() { const [count, setCount] = useState(0); const handleReset = () => setCount(0); const handleIncrement = () => setCount((prev) => prev + 1); const triggerRecordedError = () => { try { throw new Error("Simulated controlled error from Plus One app") } catch (e) { alert("You intentionally threw an error") } }; return ( <View style={styles.container}> <View style={styles.header}> <Text style={styles.headerText}>Plus One</Text> </View> <View style={styles.counterWrapper}> <Text style={styles.counterText}>{count}</Text> </View> <View style={styles.actionsRow}> <ButtonBox label="Reset" onPress={handleReset} /> <ButtonBox label="+1" onPress={handleIncrement} /> <ButtonBox label="Error" onPress={triggerRecordedError} /> </View> </View> ); } type ButtonBoxProps = { label: string; onPress: () => void; }; function ButtonBox({ label, onPress }: ButtonBoxProps) { return ( <TouchableOpacity onPress={onPress} style={styles.button} activeOpacity={0.8}> <Text style={styles.buttonText}>{label}</Text> </TouchableOpacity> ); } /* The rest of the application code */ Now that we have verified a working app, we can add observability support by downloading the observability React Native SDK. Install LaunchDarkly SDK dependencies. JavaScript npm install @launchdarkly/react-native-client-sdk npm install @launchdarkly/observability-react-native Next, you’ll need to initialize the React Native LD client in the app/_layout file. Replace the in the layout file by pasting the following code. JavaScript // app/_layout.tsx import { Observability } from '@launchdarkly/observability-react-native'; import { AutoEnvAttributes, LDOptions, LDProvider, ReactNativeLDClient } from '@launchdarkly/react-native-client-sdk'; import { Stack } from 'expo-router'; import { useEffect, useState } from 'react'; const options: LDOptions = { applicationInfo: { id: 'Plus-One', name: 'Sample Application', version: '1.0.0', versionName: 'v1', }, debug: true, plugins: [ new Observability({ serviceName: 'my-react-native-app', serviceVersion: '1.0.0', }) ], }; const userContext = { kind: 'user', key: 'test-hello' }; export default function RootLayout() { const [client, setClient] = useState<ReactNativeLDClient | null>(null); useEffect(() => { // Initialize client const featureClient = new ReactNativeLDClient( 'mob-abc123', AutoEnvAttributes.Enabled, options, ); featureClient.identify(userContext).catch((e: any) => console.log(e)); setClient(featureClient); // Cleanup function that runs when component unmounts return () => { featureClient.close(); }; }, []); if (!client) { return null; } return ( <LDProvider client={client}> <Stack /> </LDProvider> ); } First, we’re importing the Observability SDK as well as a few LD libraries to add options and attributes to the LD client. Initialized the SDK and plugin options.Defined the user context.Lastly, you initialized the client. Now that you have defined your LD React Native client, you can implement different observability methods within your application logic. We can do this by importing the LDObserve library in the app/_layout.tsx file. JavaScript import { LDObserve } from '@launchdarkly/observability-react-native'; Then, add the recordError() method within the triggerRecordedError function inside the app/_layout.tsx file. This will allow for error messages to be sent back to the LD UI. JavaScript const triggerRecordedError = () => { try { throw new Error("Simulated controlled error from Plus One app") } catch (e) { LDObserve.recordError(e as Error, {feature: "test-button"}) alert("You intentionally threw an error") } }; Before being able to receive data in the LD UI, you’ll need to add your mobile key to the React Native LD client, which can be found by logging in to the LD UI. Once logged in, tap the settings button at the bottom left. Navigate to the Projects page and click Create to create a new project. Define the new Project and click Create Project. Then, define the environment where you would like your data to be sent. Now, grab the mobile key by pressing the three dots for the environment and selecting the mobile key, which will copy the key to your keyboard. Then, add it to the app/_layout file. JavaScript const featureClient = new ReactNativeLDClient( ‘mob-abc123’, AutoEnvAttributes.Enabled, options, ); Finally, you can generate data by interacting with your app in the iOS app simulator. Feel free to restart the app to ensure data is displaying in real time. JavaScript npm expo run:ios Once you navigate back to the LD UI, you should be able to see the logs, traces, and errors under the Monitor section. Logs Traces Errors Conclusion In just a few minutes, we’ve taken the PlusOne React Native app from a simple counter to a fully observable application connected to LaunchDarkly. By setting up the SDK, initializing observability plugins, and recording errors, we now have a live feedback loop where application behavior is visible in the LaunchDarkly UI. This makes it far easier to diagnose issues, validate feature flag rollouts, and ensure smooth user experiences. Next Steps Looking ahead, there are many ways to expand on what we’ve built by including features like recording custom metrics and session replay, which provide even deeper insights into app behavior. By integrating observability at the foundation of your React Native projects, you equip your team with the clarity needed to debug faster, ship features more confidently, and deliver reliable experiences to your users. You can also read this article to learn more about observability and guarded releases.

Streaming systems usually fail in one of two ways: Loudly, when infrastructure breaksQuietly, when one bad record keeps replaying until the pipeline is effectively dead The second failure mode is more dangerous because it often starts with something small: malformed JSON, an unexpected schema change, a missing required field, or a downstream timeout that was never handled correctly. In Apache Flink, one unhandled exception can trigger a restart. If the same poison message is still sitting in Kafka after recovery, the job reads it again, fails again, restarts again, and enters a loop. At that point, the pipeline is technically "recovering," but operationally it is down. This is exactly why production Flink jobs need a Dead Letter Queue (DLQ) strategy from day one. A proper DLQ pattern does three things: Isolates bad records so they do not stop good onesCaptures enough failure context to debug the issue laterPreserves replayability so quarantined records can be reprocessed after the root cause is fixed Anything less is not really a DLQ. It is either silent data loss or delayed outage. In this article, I will walk through the most practical DLQ patterns for Apache Flink 1.18: Side outputs as the core DLQ primitiveRetry with exponential backoff for transient failuresTiered DLQ routing by error classKafka and S3 sink patternsMetrics and alertingReplay with a dedicated reprocessing jobA PyFlink version of the side output pattern The goal is simple: a bad message should never silently disappear, and it should never silently stop the stream. Why Poison Messages Break Otherwise Healthy Pipelines A poison message is any record that consistently fails processing. Typical examples include: Malformed JSONIncompatible schema versionsMissing required fieldsInvalid business valuesRecords that trigger unexpected code pathsMessages that repeatedly fail downstream enrichment calls Without DLQ handling, the failure path usually looks like this: The record enters the pipelineDeserialization or validation throws an exceptionThe operator failsFlink restarts from the last checkpointThe same record is consumed againThe same exception happens again That loop can continue indefinitely. The result is predictable: Throughput drops to zeroDownstream consumers starveCheckpoint recovery does not helpOn-call engineers get paged for a problem caused by one record This is why DLQ handling is not just an error-handling convenience. It is a core reliability pattern. What a DLQ Should Look Like in Flink In a streaming architecture, a DLQ is a durable destination for records that could not be processed successfully. For Flink, that means the DLQ record should usually include: Raw payloadError typeError messageStack trace or summarized failure contextFailure timestampSource metadata such as topic, partition, or offset when available That information matters because a DLQ is only useful if someone can answer two questions later: Why did this record fail?How do I replay it safely once the issue is fixed? If you only log the exception, you lose replayability. If you only store the payload, you lose debugging context. If you drop the record entirely, you lose both. So the design target is not "catch exceptions." The design target is durable, observable, replayable failure handling. Pattern 1: Use Side Outputs as the Core DLQ Primitive The most natural DLQ mechanism in Flink is the side output. A side output allows one operator to emit records to multiple streams: The main stream for successful recordsOne or more side streams for failures, late data, or quarantined records That makes it the right primitive for DLQ routing. Define the DLQ Envelope and Output Tag Java import org.apache.flink.util.OutputTag; import org.apache.flink.streaming.api.functions.ProcessFunction; import org.apache.flink.util.Collector; public static final OutputTag<DeadLetterRecord> DLQ_TAG = new OutputTag<DeadLetterRecord>("dead-letter-queue") {}; public record DeadLetterRecord( String rawPayload, String errorType, String errorMessage, String stackTrace, long failedAtEpochMs, String sourceTopicPartition, long sourceOffset ) {} The important point here is that the DLQ record is not just the failed payload. It is an envelope that preserves enough context for triage and replay. Route Failures Inside a ProcessFunction Java public class EntityEventProcessor extends ProcessFunction<String, EntityEvent> { @Override public void processElement( String rawMessage, Context ctx, Collector<EntityEvent> out) { try { EntityEvent event = parseAndValidate(rawMessage); out.collect(event); } catch (JsonParseException e) { ctx.output(DLQ_TAG, new DeadLetterRecord( rawMessage, "JSON_PARSE_FAILURE", e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.element().toString(), -1L )); } catch (SchemaValidationException e) { ctx.output(DLQ_TAG, new DeadLetterRecord( rawMessage, "SCHEMA_VALIDATION_FAILURE", e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.element().toString(), -1L )); } catch (Exception e) { ctx.output(DLQ_TAG, new DeadLetterRecord( rawMessage, "UNKNOWN_FAILURE", e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.element().toString(), -1L )); } } private EntityEvent parseAndValidate(String raw) throws JsonParseException, SchemaValidationException { EntityEvent event = objectMapper.readValue(raw, EntityEvent.class); if (event.entityId() == null || event.entityId().isBlank()) { throw new SchemaValidationException("entityId is required"); } if (event.timestamp() <= 0) { throw new SchemaValidationException("timestamp must be positive"); } return event; } } This is the minimum viable DLQ pattern, and it already solves the most important operational problem: bad records no longer stop good ones. Wire the Main Stream and DLQ Stream Java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<String> kafkaSource = env .fromSource(buildKafkaSource(), WatermarkStrategy.noWatermarks(), "entity-events-source"); SingleOutputStreamOperator<EntityEvent> processed = kafkaSource.process(new EntityEventProcessor()); DataStream<EntityEvent> goodEvents = processed; DataStream<DeadLetterRecord> deadLetters = processed.getSideOutput(DLQ_TAG); goodEvents.sinkTo(buildDownstreamKafkaSink()); deadLetters.sinkTo(buildDlqKafkaSink()); env.execute("Entity Resolution Pipeline"); If you do nothing else, do this. Side outputs should be the default DLQ foundation in Flink. Pattern 2: Retry Transient Failures Before Escalating to DLQ Not every failure belongs in the DLQ immediately. Some failures are transient: A downstream service is temporarily unavailableA database call times outAn external API is rate-limitedA network dependency is briefly unstable If you send all of those directly to the DLQ, you create noise and bury the truly bad records. The better pattern is: Retry transient failures a limited number of timesUse exponential backoffEscalate to DLQ only after retries are exhausted Retry With KeyedProcessFunction and Timers Java public class RetryingEnrichmentProcessor extends KeyedProcessFunction<String, EntityEvent, EnrichedEvent> { private static final int MAX_RETRIES = 3; private static final long BASE_BACKOFF_MS = 500L; private transient ValueState<Integer> retryCountState; private transient ValueState<EntityEvent> pendingEventState; @Override public void open(Configuration parameters) { retryCountState = getRuntimeContext().getState( new ValueStateDescriptor<>("retry-count", Integer.class)); pendingEventState = getRuntimeContext().getState( new ValueStateDescriptor<>("pending-event", EntityEvent.class)); } @Override public void processElement( EntityEvent event, Context ctx, Collector<EnrichedEvent> out) throws Exception { try { EnrichedEvent enriched = callEnrichmentService(event); retryCountState.clear(); pendingEventState.clear(); out.collect(enriched); } catch (TransientServiceException e) { int retries = retryCountState.value() == null ? 0 : retryCountState.value(); if (retries >= MAX_RETRIES) { retryCountState.clear(); pendingEventState.clear(); ctx.output(DLQ_TAG, new DeadLetterRecord( event.toString(), "MAX_RETRIES_EXCEEDED", "Failed after " + MAX_RETRIES + " retries: " + e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.getCurrentKey(), -1L )); } else { retryCountState.update(retries + 1); pendingEventState.update(event); long backoffMs = BASE_BACKOFF_MS * (long) Math.pow(2, retries); ctx.timerService().registerProcessingTimeTimer( System.currentTimeMillis() + backoffMs ); } } catch (PoisonMessageException e) { ctx.output(DLQ_TAG, new DeadLetterRecord( event.toString(), "POISON_MESSAGE", e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.getCurrentKey(), -1L )); } } @Override public void onTimer( long timestamp, OnTimerContext ctx, Collector<EnrichedEvent> out) throws Exception { EntityEvent pending = pendingEventState.value(); if (pending == null) return; try { EnrichedEvent enriched = callEnrichmentService(pending); retryCountState.clear(); pendingEventState.clear(); out.collect(enriched); } catch (TransientServiceException e) { int retries = retryCountState.value(); if (retries >= MAX_RETRIES) { retryCountState.clear(); pendingEventState.clear(); ctx.output(DLQ_TAG, new DeadLetterRecord( pending.toString(), "MAX_RETRIES_EXCEEDED", "Timer retry exhausted: " + e.getMessage(), getStackTrace(e), System.currentTimeMillis(), ctx.getCurrentKey(), -1L )); } else { retryCountState.update(retries + 1); long backoffMs = BASE_BACKOFF_MS * (long) Math.pow(2, retries); ctx.timerService().registerProcessingTimeTimer( timestamp + backoffMs ); } } } } Why This Works Especially Well in Flink This pattern is stronger in Flink than in many other stream processors because timers and state are checkpointed. That means: Retry counters survive restartsPending events survive restartsScheduled retries resume after recovery In other words, the retry workflow itself is fault-tolerant. That is exactly what you want when handling transient failures in a long-running stream. Pattern 3: Split the DLQ by Failure Type Once a pipeline matures, a single DLQ topic usually becomes too coarse. Schema failures, business validation failures, exhausted retries, and unknown exceptions all end up mixed together. That makes triage slower and replay harder. A better pattern is to classify failures and route them to separate DLQ streams. Define Failure Tiers Java public enum DlqTier { TRANSIENT_EXHAUSTED, SCHEMA_INVALID, BUSINESS_RULE, UNKNOWN } Route by Exception Class Java public class TieredDlqRouter extends ProcessFunction<String, EntityEvent> { @Override public void processElement( String raw, Context ctx, Collector<EntityEvent> out) { try { EntityEvent event = parse(raw); validate(event); out.collect(event); } catch (JsonParseException | MappingException e) { route(ctx, raw, DlqTier.SCHEMA_INVALID, e); } catch (BusinessValidationException e) { route(ctx, raw, DlqTier.BUSINESS_RULE, e); } catch (Exception e) { route(ctx, raw, DlqTier.UNKNOWN, e); } } private void route(Context ctx, String raw, DlqTier tier, Exception e) { OutputTag<DeadLetterRecord> tag = getTierTag(tier); ctx.output(tag, new DeadLetterRecord( raw, tier.name(), e.getMessage(), getStackTrace(e), System.currentTimeMillis(), "", -1L )); } } Define One Output Tag Per Tier Java public static final OutputTag<DeadLetterRecord> DLQ_SCHEMA = new OutputTag<>("dlq-schema-invalid") {}; public static final OutputTag<DeadLetterRecord> DLQ_BUSINESS = new OutputTag<>("dlq-business-rule") {}; public static final OutputTag<DeadLetterRecord> DLQ_UNKNOWN = new OutputTag<>("dlq-unknown") {}; Sink Each Tier Independently Java SingleOutputStreamOperator<EntityEvent> processed = kafkaSource.process(new TieredDlqRouter()); processed.getSideOutput(DLQ_SCHEMA) .sinkTo(buildKafkaSink("dlq.schema-invalid")); processed.getSideOutput(DLQ_BUSINESS) .sinkTo(buildKafkaSink("dlq.business-rule")); processed.getSideOutput(DLQ_UNKNOWN) .sinkTo(buildKafkaSink("dlq.unknown")); This makes the DLQ operationally useful instead of just technically correct. For example: Schema failures can be routed to the producer teamBusiness rule failures can feed data quality workflowsUnknown failures can trigger higher-severity alerting Pattern 4: Choose DLQ Sinks Based on How You Plan To Recover Once records are routed to a DLQ stream, they need a durable destination. In practice, the two most common choices are Kafka and object storage. Kafka DLQ Sink Kafka is the right choice when you want: Near-real-time inspectionStreaming replayOperational integration with existing consumers Java private static KafkaSink<DeadLetterRecord> buildDlqKafkaSink( String topicName) { return KafkaSink.<DeadLetterRecord>builder() .setBootstrapServers("kafka-broker:9092") .setRecordSerializer( KafkaRecordSerializationSchema.builder() .setTopic(topicName) .setValueSerializationSchema( new JsonSerializationSchema<>(DeadLetterRecord.class)) .setKeySerializationSchema( record -> record.errorType().getBytes()) .build() ) .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE) .build(); } S3 DLQ Sink Object storage is the better choice when you want: Long retentionLow-cost quarantineBatch replay with Spark or AthenaPartitioned storage by date or error type Java private static FileSink<DeadLetterRecord> buildS3DlqSink() { return FileSink .forRowFormat( new Path("s3://your-bucket/dlq/entity-resolution/"), new JsonRowEncoder<>(DeadLetterRecord.class) ) .withRollingPolicy( DefaultRollingPolicy.builder() .withRolloverInterval(Duration.ofMinutes(15)) .withInactivityInterval(Duration.ofMinutes(5)) .withMaxPartSize(MemorySize.ofMebiBytes(128)) .build() ) .withBucketAssigner( new DateTimeBucketAssigner<>( "error-type='unknown'/year=yyyy/month=MM/day=dd/hour=HH") ) .build(); } A practical production pattern is to use: Kafka for short-term operational handlingS3 for long-term quarantine and replay That gives you both fast response and durable history. Pattern 5: Monitor DLQ Rate, Not Just Job Uptime A DLQ that nobody watches is just a backlog with better branding. Job uptime alone is not enough. A Flink job can stay green while quietly routing 10% of traffic to the DLQ. That is still a production incident. Add Metrics Inside the Operator Java public class MonitoredEntityEventProcessor extends ProcessFunction<String, EntityEvent> { private transient Counter dlqCounter; private transient Counter successCounter; private transient Histogram processingLatency; @Override public void open(Configuration parameters) { MetricGroup metrics = getRuntimeContext() .getMetricGroup() .addGroup("entity_resolution"); dlqCounter = metrics.counter("dlq_routed_total"); successCounter = metrics.counter("processed_success_total"); processingLatency = metrics.histogram( "processing_latency_ms", new DescriptiveStatisticsHistogram(1000) ); } @Override public void processElement( String raw, Context ctx, Collector<EntityEvent> out) { long start = System.currentTimeMillis(); try { EntityEvent event = parseAndValidate(raw); successCounter.inc(); out.collect(event); } catch (Exception e) { dlqCounter.inc(); ctx.output(DLQ_TAG, buildDeadLetter(raw, e)); } finally { processingLatency.update(System.currentTimeMillis() - start); } } } Alert on DLQ Rate A useful alert is DLQ throughput relative to successful throughput: YAML - alert: FlinkDlqRateHigh expr: | rate(flink_entity_resolution_dlq_routed_total[5m]) / rate(flink_entity_resolution_processed_success_total[5m]) > 0.01 for: 2m labels: severity: warning annotations: summary: "DLQ rate exceeds 1% of total throughput" description: "Check dlq.unknown Kafka topic for upstream schema changes" As a rule of thumb: above 1% often indicates schema drift or producer issuesabove 5% usually indicates a broader systemic problem The exact thresholds depend on the pipeline, but the principle does not: monitor DLQ rate as a first-class health signal. Pattern 6: Replay With a Dedicated Reprocessing Job A DLQ is only complete when replay is possible. The cleanest design is a separate Flink job that reads from the DLQ topic and routes records back through the main processing logic. Example Replay Job Java public class DlqReprocessingJob { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<DeadLetterRecord> dlqStream = env .fromSource( buildKafkaSource("dlq.schema-invalid"), WatermarkStrategy.noWatermarks(), "dlq-source" ); DataStream<String> replayStream = dlqStream .filter(r -> r.failedAtEpochMs() >= START_EPOCH && r.failedAtEpochMs() <= END_EPOCH) .map(DeadLetterRecord::rawPayload); SingleOutputStreamOperator<EntityEvent> reprocessed = replayStream.process(new EntityEventProcessor()); reprocessed.sinkTo(buildDownstreamKafkaSink()); reprocessed.getSideOutput(DLQ_TAG) .sinkTo(buildKafkaSink("dlq.permanent-quarantine")); env.execute("DLQ Reprocessing Job"); } } Why Replay Should Be a Separate Job Keeping replay separate from the main pipeline gives you: Independent scalingIndependent schedulingCleaner checkpoint behaviorSafer operational control It also lets you drain backlogs on your own terms: Off-peak hoursReduced parallelismOr maximum parallelism when you need to catch up quickly That separation keeps the main pipeline stable while still making recovery practical. PyFlink Version: Same Pattern, Same Principle If your team uses PyFlink, the same side output pattern applies. Python from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.functions import ProcessFunction from pyflink.common.typeinfo import Types from pyflink.datastream.output_tag import OutputTag DLQ_TAG = OutputTag( "dead-letter-queue", Types.ROW_NAMED( ["raw_payload", "error_type", "error_message", "failed_at_ms"], [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG()] ) ) class EntityEventProcessor(ProcessFunction): def process_element(self, value, ctx): try: event = parse_and_validate(value) yield event except Exception as e: from pyflink.common import Row yield DLQ_TAG, Row( raw_payload=str(value), error_type=type(e).__name__, error_message=str(e), failed_at_ms=int(time.time() * 1000) ) env = StreamExecutionEnvironment.get_execution_environment() source_stream = env.from_source(...) processed = source_stream.process( EntityEventProcessor(), output_type=Types.STRING() ) good_events = processed dead_letters = processed.get_side_output(DLQ_TAG) good_events.sink_to(build_downstream_sink()) dead_letters.sink_to(build_dlq_sink()) env.execute("Entity Resolution Pipeline") The syntax changes, but the design principle stays the same: good records continue, bad records are isolated and persisted. Production Checklist Before shipping a Flink pipeline, verify the following: RequirementWhy It MattersRisky operators wrapped in try/catchPrevents restart loops from unhandled exceptionsDLQ output tags use explicit typingAvoids runtime serialization failuresDLQ sink is durableFailed records must survive restartsDLQ metrics are exportedSilent DLQ growth is otherwise invisibleReplay path exists and is testedA DLQ without replay is just storageDLQ retention is long enoughTeams need time to diagnose and replayPermanent quarantine existsPrevents infinite replay loopsAlerting is based on DLQ rateJob health alone is not enough This checklist is worth automating in code review or deployment readiness checks. DLQ handling is too important to leave to convention. Key Takeaways If you are building Flink pipelines in production, the safest default is: Use side outputs for DLQ routingRetry transient failures before escalationClassify failures into separate DLQ streamsSink DLQ records durablyExport DLQ metricsReplay through a dedicated job The core rule is simple: A bad message should never silently disappear, and it should never silently stop the stream. That is what turns DLQ handling from a defensive coding trick into a real reliability pattern. Environment Notes The examples in this article target: Apache Flink 1.18Java 17PyFlink 1.18 A few implementation notes: The retry timer pattern requires a keyed stream before KeyedProcessFunctionRocksDB is usually the safer state backend for larger retry stateHashMap state backend can work well for smaller, latency-sensitive workloadsAT_LEAST_ONCE is usually sufficient for DLQ sinks Final Thoughts Poison messages are not rare in streaming systems. They are inevitable. The real question is whether one bad record can take down an otherwise healthy pipeline. With the right DLQ design in Flink, the answer becomes no. The stream keeps moving. Good records continue. Bad records are quarantined. Alerts fire. Replay remains possible. And the pipeline stays operational while the root cause is fixed. That is the difference between a stream that works in staging and one that survives production.