Databases

Most vehicle tracking systems ask one database to do everything. For example, store the road network, query it with recursive CTEs, write live positions to the same database, run analytics on the same tables, and so on. It works until the graph queries slow down, high-frequency writes start competing with reads, and analytics queries time out. This article shows a different approach: three databases, each doing what it's genuinely good at. Neo4j Aura for the road network graph, Databricks Lakebase for live vehicle positions, and Databricks Lakehouse for historical analytics. Ten simulated vehicles move around a real city following real road connections loaded from OpenStreetMap. Two Streamlit dashboards show live positions and analytics. The whole system is driven by a single YAML configuration file, so switching from London to San Francisco or Singapore means changing a single file and rerunning five notebooks. The full source code is available on GitHub. The Three-System Architecture The architecture has three distinct layers: Neo4j Aura holds the road network — intersections, road segments, zone topology, and shortest paths. It answers graph questions that a relational database may handle awkwardly.Databricks Lakebase holds the live operational data — vehicle positions written every two seconds by a simulator, vehicle statuses, and trip records. It's a fully managed Postgres database inside Databricks, handling OLTP workloads with standard psycopg2 connectivity.Databricks Lakehouse holds the analytical history — position data synced from Lakebase into a Delta table, aggregated by zone and road segment. None of these systems knows about the others. The intelligence sits in the application layer — the simulator, the Streamlit dashboards and the analytics notebook — which orchestrates queries across all three and combines the results. The Road Network in Aura We'll use OSMnx to download the drivable road network for the London Borough of Merton from OpenStreetMap and load it into Aura. The graph model is straightforward: Cypher (:Intersection {node_id, lat, lon, street_count, location}) -[:ROAD {osmid, name, highway, maxspeed, oneway, length_m}]-> (:Intersection) Merton's road network produces thousands of intersection nodes and thousands of directed road relationships. A POINT INDEX on the location property enables fast nearest-neighbor lookups -- finding the intersection closest to any GPS coordinate runs in milliseconds. The reason for Aura is simple: the road network is a graph and graph queries are where Aura excels. Finding the shortest path between two zones is a single Cypher function call: Cypher MATCH path = shortestPath((start)-[:ROAD*..300]->(end)) RETURN length(path) AS hops The equivalent in SQL requires a recursive CTE that grows in complexity with every additional hop. For zone reachability queries — "which zones can a vehicle reach within two hops?" — the difference is even more pronounced. We also define five logical zones as bounding boxes within the borough and store them as Zone nodes with ADJACENT_TO relationships. This gives us a zone adjacency graph that the simulator uses for routing decisions. The Simulator The simulator loads the entire road graph from Aura into memory at startup — one query, one dictionary, no further Aura calls during the simulation loop. It then places ten vehicles at their home intersections and moves each one along Breadth-First Search (BFS)-computed routes. Vehicles don't route randomly. They have a home zone and a 70% chance of staying in or near it. The remaining 30% of the time, they cross into any zone in the borough, producing occasional longer cross-city runs. Every two seconds, each vehicle writes its current coordinates to Lakebase: Python cursor.execute(""" INSERT INTO vehicle_positions (vehicle_id, lat, lon, speed_kmh, current_zone) VALUES (%s, %s, %s, %s, %s) """, (vehicle_id, lat, lon, speed_kmh, current_zone)) The simulator runs as a background subprocess launched from a Jupyter notebook, continuing independently while the Streamlit dashboards are open. The Live Vehicle Tracker The vehicle tracker (app.py) refreshes every three seconds and shows three pydeck layers on a CARTO basemap: Vehicle icons – one car icon per vehicle at its current GPS positionTrail lines – each vehicle's last 20 positions, colored by home zoneShortest path – a black line showing the road-network shortest path between any two selected zones, computed on demand from Aura Figure 1 shows vehicles moving on the Merton map with trail lines and a shortest path highlighted between two zones. Figure 1. Streamlit Vehicle Tracker. The sidebar shows a bar chart of zone activity over the last 10 minutes and a nearest-driver lookup -- given a zone, which vehicle is currently closest to it? The haversine distance calculation runs against the latest position of every vehicle, using zone center coordinates that map to real road intersections. The Analytics Dashboard The analytics dashboard (analytics_app.py) connects to all three systems simultaneously. Every 30 seconds, it syncs new position records from Lakebase into a Lakehouse Delta table and runs two analytical queries. Figure 2 shows an analytics dashboard with zone activity over time across all five zones. Figure 2. Analytics Dashboard. The chart on the left-hand side shows position update counts per zone per minute over the last hour — a live view of which parts of the city are busiest: SQL SELECT current_zone AS zone, DATE_TRUNC('minute', recorded_at) AS minute, COUNT(*) AS updates FROM vehicle_positions_delta WHERE current_zone IS NOT NULL GROUP BY current_zone, DATE_TRUNC('minute', recorded_at) ORDER BY minute, zone The chart on the right-hand side is the architectural highlight: a cross-system join that answers "which named roads carry the most vehicle traffic?" Lakebase has the position records (latitude, longitude, per vehicle per tick). Aura has the road names (what named road each intersection belongs to). Neither system alone can answer the question. The join runs in Python using pandas. Road names and coordinates are loaded from Aura once at startup and cached. Position coordinates come from Lakebase via the Lakehouse Delta table on each refresh. Coordinates are rounded to three decimal places (~100m precision) and joined: Python joined = pos_df.merge( road_df[["road_name", "highway", "lat_r", "lon_r"]], on=["lat_r", "lon_r"], how="inner" ) Primary roads dominate because BFS routing naturally follows main roads when finding shortest paths. The YAML Configuration System Every city-specific value lives in a single config.yaml file which contains zone definitions, vehicle assignments, map coordinates, and the OpenStreetMap place name. YAML city: name: "London Borough of Merton" osmnx_place: "London Borough of Merton, UK" network_type: "drive" map_lat: 51.410 map_lon: -0.188 map_zoom: 12 Switching cities means copying a different config file and re-running five notebooks. Three example config files are included: Merton (London), San Francisco, and Singapore. For cities where OSMnx's place name geocoding doesn't produce a usable polygon boundary, a pyrosm-based approach clips a Geofabrik regional file to a bounding box instead. The pre-clipped files for San Francisco and Singapore are included in the GitHub repo, so you can run those configs without any additional data preparation. A companion config_validator.py validates the file on load and raises clear errors if anything is missing or malformed. Why Three Systems? The answer is that each system does something the others can't do efficiently. Neo4j Aura handles graph traversals — shortest paths, multi-hop reachability, nearest-node spatial lookups. These are awkward in SQL and natural in Cypher. Databricks Lakebase handles high-frequency OLTP writes — hundreds of inserts per minute, sustained, with foreign key constraints and BIGSERIAL auto-increment. Databricks Lakehouse handles analytical aggregations over historical data — counting position records by zone and minute, joining across large datasets. Columnar storage and parallel execution make this fast. The three-system architecture isn't complexity for its own sake. Each system earns its place by doing something the others would handle poorly. The Free Online Book The full system — all notebooks, both Streamlit apps, the YAML config system and seven chapters of detailed explanation — is available as a free online book. The book covers the road network loading and data cleaning, zone and adjacency graph setup, Lakebase table design, the BFS simulator, both Streamlit dashboards, the analytics notebook, and all the gotchas and lessons learned. The code is on GitHub under Apache 2.0. The pre-clipped OSM data files are available under the Open Database License (ODbL). Summary We've built a real-time fleet operations dashboard using three database systems, each doing what it does best: Neo4j Aura for road network graph queries and shortest path computation, Databricks Lakebase for high-frequency vehicle position writes, and Databricks Lakehouse for historical analytics over Delta tables. The interesting engineering is in the joins that cross system boundaries — finding the nearest driver uses Aura's spatial index, routing vehicles uses BFS over an in-memory graph loaded from Aura, and identifying the busiest named roads joins position data with road names via pandas. A YAML configuration file drives the entire system, making it straightforward to point the same codebase at a different city. The architecture demonstrates that a multi-database approach isn't inherently complex — it becomes simpler when each system has a clear, non-overlapping role. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Data engineers managing batch SQL pipelines on Snowflake, BigQuery, and increasingly Databricks, and streaming pipelines on Apache Flink face a familiar problem: two toolchains, two skill sets, two CI/CD pipelines.dbt is now extending into stream processing. This post explains what that means in practice, why it matters for data engineering teams, and what a concrete implementation looks like with Apache Flink on Confluent Cloud. Data Streaming Meets the Lakehouse Data lakes promised to solve the enterprise data problem. The reality has been messier. Batch pipelines produce stale information, and analytical workloads run hours after the business event occurred. By the time a query runs, the window for action is often already closed. The lakehouse pattern has improved matters. Apache Iceberg has become the dominant open table format, supported across Snowflake, Databricks, BigQuery, and a growing number of query engines. Teams can run SQL analytics directly on data in object storage without duplicating it into a proprietary warehouse. But the lakehouse alone does not solve the real-time problem. Data still arrives as a batch, minutes or hours after the source event. That gap reflects a deeper architectural split. Data streaming with Apache Kafka and Flink is the operational layer: it handles critical SLAs, powers event-driven applications, and keeps business systems running in real time. The lakehouse is the analytical layer: it stores historical data for reporting, ML, and near real-time or batch analytics. These are two distinct workloads with different requirements regarding uptime, data loss, latency, and throughput. They need to coexist without forcing engineers to build and maintain two separate pipelines. How Kafka, Flink, and Iceberg Work Together That is what the combination of Apache Kafka, Apache Flink, and Apache Iceberg addresses. Kafka captures every event at the source and serves as the operational backbone for real-time systems. Flink processes and enriches data in motion, supporting both immediate operational decisions and the preparation of data for downstream analytics. Iceberg stores the result as a governed, queryable table for any analytical engine, whether that is Snowflake, BigQuery, or Databricks. A full treatment of this architecture, including schema evolution, compaction, and catalog integration, is covered here: Data Streaming Meets Lakehouse: Apache Iceberg for Unified Real-Time and Batch Analytics. The question is no longer whether streaming and lakehouse architectures can coexist. They already do. The question is how data engineering teams can work across both without maintaining separate toolchains. That is where dbt enters the picture. What Is dbt? dbt, the data build tool, is an open-source framework for SQL-based data transformation. A dbt model is a SQL SELECT statement saved as a file. dbt infers execution order from how models reference each other using ref(). The standard commands cover the full engineering workflow: dbt run executes the SQL against the target platform, dbt test validates data quality, and dbt docs generate produces a browsable documentation catalog. What made dbt successful is the discipline it brings to SQL work. Before dbt, transformation logic lived in scattered scripts and proprietary ETL tools. dbt replaced that with a code-first, version-controlled workflow with built-in lineage, testing, and documentation. Snowflake and BigQuery are where most dbt adoption lives today. Both are SQL-native and optimized for the ELT pattern dbt was built around. Redshift is a strong third platform in AWS environments. Databricks has seen growing dbt adoption more recently, driven by investments in serverless SQL Warehousing, but its roots are in Spark and Python, making it a newer entrant in the dbt ecosystem. dbt Labs crossed $100 million in ARR in early 2025, with over 5,000 paying customers. Around 90,000 dbt projects are running in production today. The Fivetran and dbt Labs merger, announced in October 2025, created a combined data infrastructure company with nearly $600 million in annual revenue — a clear signal that dbt has moved well beyond a popular open-source tool and into foundational enterprise data infrastructure. dbt Meets Apache Flink: One Workflow for Data Engineers Data engineering teams managing both batch and streaming today operate in two separate realities. Snowflake or BigQuery on one side: dbt models, version-controlled SQL, automated tests, generated docs. Apache Flink on the other: Terraform scripts, custom deployment code, or the Flink console. Skills and practices do not transfer between the two. That separation has a real cost. Streaming pipelines are harder to test, harder to document, and harder to hand over. Many teams compensate by keeping streaming logic minimal and pushing transformation work downstream into the warehouse, which reintroduces latency and undermines the point of streaming. The vision is straightforward: one SQL workflow for both. The engineer who builds dbt models on Snowflake or BigQuery should be able to apply the same approach to an Apache Flink streaming pipeline, without switching tools or rebuilding CI/CD from scratch. Two toolchains mean two testing strategies, two documentation systems, and two skill sets to hire and retain. Governance enforcement becomes inconsistent across the two environments. SQL is the shared foundation that makes this realistic. Flink SQL is mature and production-proven. Snowflake and BigQuery are SQL-native. Apache Iceberg tables are queryable via SQL across multiple engines. dbt wraps SQL with engineering discipline. The model files look the same. The ref() dependency resolution works the same way. Tests and documentation generation work through the same commands. Organizations do not need to hire separate Flink infrastructure specialists. The existing data engineering team can own both sides. Apache Iceberg connects the two worlds at the storage layer. A Flink pipeline writes structured, governed events into an Iceberg table in the organization's own S3 bucket. That same table is immediately readable by Snowflake, BigQuery, or Databricks without any additional ETL step. dbt can model data across the full pipeline: shaping it as it streams through Flink, and transforming it again when it lands in the warehouse for analytics. This is also a direct enabler of the Shift Left Architecture 2.0. The Shift Left approach moves data integration logic closer to the source, applying quality checks, enrichment, and governance in the streaming layer before data lands in the lakehouse. Until now, that required streaming-specific skills that most dbt-native teams did not have. dbt for Flink lowers that barrier considerably. The full architectural detail is covered here: The Shift Left Architecture 2.0: Operational, Analytical and AI Interfaces for Real-Time Data Products. Concrete Example: dbt on Confluent Cloud with Apache Flink The most concrete implementation available today is the dbt-confluent adapter, released by Confluent alongside the confluent-sql Python driver. Both are open source and available on PyPI and GitHub. Data engineers define streaming pipelines as dbt models and deploy them to Flink compute pools using the standard dbt run command. Getting started is a single step: pip install dbt-confluent Three materializations are supported: view for a virtual Flink SQL view over a Kafka topic, streaming_table for a continuous always-current result set, and streaming_source for defining a Kafka topic as a dbt source. Testing is deterministic, using Confluent Cloud's snapshot query capability to return bounded point-in-time results rather than silently passing on timeout. Documentation generation works through INFORMATION_SCHEMA integration, producing the same browsable catalog that Snowflake and BigQuery projects generate. The underlying confluent-sql driver is DB-API v2 compliant, meaning any compatible tool can connect directly to Confluent Cloud Flink: Airflow and Dagster for orchestration, Pandas for snapshot queries, Streamlit for live dashboards, and LangChain for AI agent workflows. For data engineers already working in dbt, this means the skills and practices built around Snowflake or BigQuery transfer directly to the streaming side of the architecture. The Data Engineer Owns Batch and Streaming with dbt The separation between batch and streaming engineering has always been more organizational than technical. Both worlds use SQL. Both require testing, documentation, and reliable deployment. The tools just never bridged the gap, so organizations staffed and operated two distinct engineering disciplines. dbt extending to Apache Flink changes that equation. The data engineer who runs dbt on Snowflake or BigQuery today can apply the same mental model, commands, and CI/CD pipeline to Flink streaming pipelines. No Flink infrastructure specialization required. They write SQL models, define tests, generate documentation, and deploy, exactly as they do for batch. The implication is straightforward. The investment in dbt skills and tooling now extends further into the architecture. Streaming can be adopted incrementally by the same data engineering teams already trusted for batch. One team, one tool, one governance standard, across both operational and analytical workloads. The Flink adapter for dbt is earlier in maturity compared to dbt on Snowflake or BigQuery, and teams should expect to work with an evolving ecosystem. But the foundation is solid, the direction is clear, and the core architectural components are already running in production at scale across multiple industries. The demand from data engineering teams is real and growing.

By Kai Wähner CORE

As Large Language Models (LLMs) become increasingly integrated into enterprise applications, optimizing response time and reducing operational costs have become critical priorities. One of the most effective techniques for achieving both is Prompt Caching. Instead of processing identical prompt segments repeatedly, prompt caching allows AI systems to reuse previously computed prompt representations, minimizing redundant computation. While tokenization converts text into tokens that the model understands, prompt caching goes a step further by reusing the processing of unchanged token sequences, resulting in faster inference, lower latency, and reduced API costs, especially in applications with repetitive system prompts or recurring contextual information. How Prompt Caching Works Think of prompt caching as a “memory shortcut” for AI models. Every prompt is first tokenized, but when the same prompt prefix appears again, the model doesn’t need to process those tokens from scratch. Instead, it retrieves the cached computation and only processes the new or modified portion of the prompt. How Prompt Caching Works This mechanism is particularly valuable in AI assistants, enterprise chatbots, coding copilots, document analysis platforms, and Retrieval-Augmented Generation (RAG) systems where a significant portion of the prompt remains unchanged across multiple requests. Best Practices to Maximize Prompt Cache Efficiency To fully leverage prompt caching, organizations should design prompts strategically. Keep system instructions consistent, place static context before dynamic user inputs, avoid unnecessary formatting changes, and modularize prompt templates. These practices increase cache hit rates, reducing both processing time and infrastructure costs. Monitoring cache performance metrics, such as cache hit ratio, latency improvements, and token savings, helps teams continuously optimize AI workloads while maintaining response quality. Business Benefits and Real-World Impact Prompt caching delivers measurable business value beyond technical optimization. Organizations can reduce AI inference costs, improve application responsiveness, support higher request volumes, and enhance the overall user experience. Development teams also benefit from more predictable performance and scalable AI architectures. As enterprise AI adoption grows, prompt caching is becoming an essential optimization technique for building efficient, reliable, and cost-effective generative AI solutions. Where Prompt Cache Is Stored: Understanding the Architecture Where a prompt cache is stored depends entirely on which level of the caching architecture you are referring to. To understand where it lives, it is helpful to divide prompt caching into its two primary forms: Provider-Native Caching (Model-Level) When you use built-in prompt caching features from providers such as OpenAI, Anthropic (Claude), Google (Gemini), or DeepSeek, the cache is managed internally within the provider’s cloud infrastructure. What is Stored The cache does not store text or responses. Instead, it stores KV Tensors (Key-Value pairs). These are the raw, mathematical attention states that the model's neural network calculated during the "prefill" phase of your prompt Where Will it Live? GPU VRAM / High-Speed RAM: Because these tensors must be accessed instantly to keep latency ultra-low, they are stored directly in the high-speed volatile memory (VRAM) of the AI chips (GPUs/TPUs) or ultra-fast host system memory in the provider's data centers. Internal Distributed Storage: Since GPU memory is highly constrained and expensive, providers use advanced, proprietary cache-eviction systems. If a cache prefix isn't used for a few minutes (the Time-to-Live or TTL), it is automatically evicted (deleted) from the GPU memory to make room for other users Who Has Access? The provider manages this entirely behind the scenes. You cannot download, inspect, or manually move these KV tensors; the system simply checks the memory automatically during your API call and applies a discount if it finds a match. Application-Level Caching (User-Controlled Layer) If you are building your own caching layer in front of the LLM API to save even more money by bypassing the LLM entirely for repeat queries, you get to choose where it is stored In-Memory Databases (Most Common) Platforms like Redis or Memcached are the industry standard. Because they store data directly in RAM, they can fetch cached prompts in microseconds Vector Databases (For Semantic Caching) If you want to detect "semantically similar" prompts (e.g., matching "How do I reset my password?" with "I forgot my password"), the cache stores the text embeddings. This is stored in vector databases like Pinecone, Milvus, Qdrant, Weaviate, or pgvector (PostgreSQL) Relational / NoSQL Databases (For Archive/Backup) Standard databases like MongoDB, DynamoDB, or PostgreSQL are used to persistently store historical prompt-response pairs, though they have slightly higher retrieval latency than Redis Building a Semantic Cache With Redis involves upgrading from traditional "exact-match" caching to vector-based similarity caching. Instead of storing raw text, you store the mathematical representation (embeddings) of prompts. When a new prompt comes in, you convert it to an embedding and ask Redis to find the "nearest neighbor" (most similar prompt). If the similarity score exceeds your defined threshold (e.g., 95% similar), it's a Cache Hit. Here is the step-by-step guide to building a semantic cache using Python, Redis Stack (which includes vector search), and an embedding model (like OpenAI's). Prerequisites Redis Stack: You must use Redis Stack (or Redis Enterprise), as standard Redis does not support vector search. You can run it locally via Docker: docker run -d -p 6379:6379 redis/redis-stack-server:latest. Python Libraries: Install the required clients. pip install redis openai numpy: Redis also has a dedicated library called redisvl (Redis Vector Library) built specifically for this, which abstracts a lot of the boilerplate. Note: Redis also has a dedicated library called redisvl (Redis Vector Library) built specifically for this, which abstracts a lot of the boilerplate. The workflow follows four steps: Embed: Convert the incoming user prompt into a vector embedding. Search: Query Redis using a K-Nearest Neighbors (KNN) vector search. Evaluate: If the highest similarity score is above your threshold (e.g., > 0.92), return the cached response. Fallback and store: If no match is found, send the prompt to the LLM, return the response to the user, and store the new embedding and response in Redis Conceptual Python Implementation How the logic flows using standard redis-py and OpenAI: Python import redis import numpy as np from openai import OpenAI from redis.commands.search.query import Query # 1. Initialize Clients redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True) openai_client = OpenAI(api_key="YOUR_API_KEY") # Configuration THRESHOLD = 0.95 # 95% similarity required for a cache hit INDEX_NAME = "prompt_cache_idx" def get_embedding(text): """Convert text to an embedding vector.""" response = openai_client.embeddings.create( input=text, model="text-embedding-3-small" ) return np.array(response.data[0].embedding, dtype=np.float32).tobytes() def check_semantic_cache(prompt_text): """Search Redis for a semantically similar prompt.""" query_vector = get_embedding(prompt_text) # Construct a KNN Vector Search Query in Redis q = Query(f"*=>[KNN 1 @prompt_vector $vec AS score]")\ .return_fields("response", "score")\ .sort_by("score")\ .dialect(2) res = redis_client.ft(INDEX_NAME).search( q, query_params={"vec": query_vector} ) if res.docs: # Redis returns distance (0 is perfect match). Convert to similarity. similarity = 1 - float(res.docs[0].score) if similarity >= THRESHOLD: print(f"✅ Cache Hit! (Similarity: {similarity:.2f})") return res.docs[0].response print("❌ Cache Miss.") return None def store_in_cache(prompt_text, llm_response): """Store the new prompt and response in Redis.""" prompt_vector = get_embedding(prompt_text) # Store as a Redis Hash doc_id = f"cache:{hash(prompt_text)}" redis_client.hset(doc_id, mapping={ "prompt": prompt_text, "response": llm_response, "prompt_vector": prompt_vector }) # Optional: Set a Time-To-Live (TTL) so the cache clears old entries redis_client.expire(doc_id, 86400) # 24 hours Best Practices for Production Use a library: Instead of writing the raw vector math and RediSearch queries yourself, use RedisVL (pip install redisvl) or LangChain's Redis Cache integration. They have built-in SemanticCache classes that handle index creation and threshold tuning with just 3 lines of code. Tune your threshold carefully: A threshold that is too low (e.g., 0.80) will cause "false positives" (returning an answer to a question that is only vaguely related). A threshold too high (e.g., 0.99) defeats the purpose, acting almost like an exact-match cache. Test with 0.92 to 0.95 as a baseline. Filter by user/tenant: If you are building a multi-tenant app, make sure to add metadata tags (like user_id or tenant_id) to your Redis hashes. Your vector query must pre-filter by the user_id, so User A doesn't accidentally get a cached response meant for User B. Cost Savings by Major Provider LLM providers apply discounts specifically to input tokens that hit the cache (output tokens are always billed at the standard rate) Real-World Impact and Key Benchmarks Enterprise scale: One of the big Tech companies, like TikTok, has reported cutting their AI agent inference costs by 50% with minimal code adjustments. Agentic architectures: For complex, long-running agentic workflows (where a system prompt and conversation history are repeatedly sent over dozens of steps), prompt caching typically achieves 78% to 81% total cost reductions because the massive system instructions only need to be processed once. Break-even point: On platforms like Anthropic (which charge a 25% premium to write to the cache), you only need to hit the cache twice on a given prompt prefix to break even and start saving money. Every subsequent read is essentially 90% off. In addition to saving money, prompt caching dramatically improves user experience by skipping the heavy "prefill" computation. It reduces Time-to-First-Token (TTFT) by 50% to 85%, meaning long documents or extensive chat histories return responses in a fraction of a second instead of causing a noticeable delay. Take Action: Build Smarter AI Applications Prompt caching is no longer an optional optimization—it’s a competitive advantage for organizations deploying AI at scale. If you’re building enterprise AI applications, evaluate where repetitive prompts exist and redesign your prompt architecture to maximize cache utilization. Small changes in prompt design can lead to significant savings in cost, latency, and compute resources.

When engineering teams build distributed systems, they naturally reach for REST over HTTP/1.1 with JSON payloads. JSON is readable, universally supported, and trivially easy to debug with any browser or proxy tool. For early-stage services handling modest traffic, that convenience is a genuine engineering asset. But as microservice topologies scale toward hundreds of nodes handling tens of thousands of concurrent requests, text-based serialization frequently evolves from a minor convenience into a measurable architectural bottleneck. CPU utilization climbs, p99 latencies widen, and intra-zone bandwidth costs quietly compound across every internal service hop. Transitioning internal service-to-service communication to Protocol Buffers (Protobuf) over HTTP/2 via gRPC is one of the most effective and high-leverage responses to this problem. This article breaks down exactly why JSON degrades at scale, how Protobuf's binary wire format addresses those root causes, and how to execute a zero-downtime migration without breaking your running services. The Hidden Cost of Text-Based Serialization at Scale To understand why JSON degrades at high throughput, you have to look past network bandwidth and examine CPU behavior directly. JSON is a text-based, schema-less format. Every time a microservice ingests a JSON payload, the runtime must allocate memory on the heap, parse raw strings, map keys to internal structs via reflection, and convert values to their respective data types. At low volumes, this parsing overhead is negligible. At enterprise scale, it compounds into a real problem across two distinct dimensions. 1. CPU-Bound Allocation and GC Churn In languages with managed memory runtimes, such as Go, Java, and Node.js being the most common in microservice architectures, parsing thousands of large JSON strings per second causes significant garbage collection pressure. Each incoming payload generates a burst of short-lived string allocations on the heap. The garbage collector is forced to run more frequently to reclaim this memory, and in runtimes that use stop-the-world collection phases, this directly spikes p99 tail latencies. The problem is not that JSON parsing is intrinsically slow on a single call. The problem is that at scale, thousands of calls per second accumulate into sustained allocation pressure that the GC cannot absorb cleanly. 2. Network Payload Bloat JSON payloads are structurally verbose because every single message must explicitly include field names as strings. Consider this representative internal service message: JSON { "transaction_id": "tx_9988112233", "account_status": "ACTIVE", "retry_count": 3 } On the wire, this payload consumes roughly 85 bytes. More than half of those bytes (over 50) are dedicated purely to transmitting key metadata: the strings "transaction_id", "account_status", and "retry_count". These keys carry no runtime information that the receiving service doesn't already know from its own code. They are structural overhead repeated on every single message. Multiply this across millions of internal RPC calls through a service mesh and you are looking at gigabytes of redundant key data transmitted intra-zone every day. That's bandwidth you are paying for and CPU cycles you are spending to parse, without gaining any informational value. The Mechanics of the Binary Shift: Why Protobuf Moves the Needle Protocol Buffers eliminate text overhead by relying on a strict Interface Definition Language (IDL) and a highly compressed binary wire format. Instead of transmitting field names, Protobuf assigns each field a unique integer tag. When a message is serialized, the keys are stripped out entirely. The wire representation of any field is just its integer tag combined with a wire type identifier, followed by the raw data bytes. The equivalent of the JSON example above looks like this as a .proto definition: ProtoBuf syntax = "proto3"; message AccountTransaction { string transaction_id = 1; string account_status = 2; int32 retry_count = 3; } The same AccountTransaction message with the values tx_9988112233, ACTIVE, and 3 serializes to approximately 24 bytes on the wire — a reduction of roughly 72% compared to the JSON equivalent. Varints and Length-Delimited Encoding Two specific encoding techniques drive most of that size reduction. Varints (Variable-Length Quantities): Standard integers occupy a fixed 4 or 8 bytes regardless of their actual value. Protobuf varints use the most significant bit as a continuation flag, meaning small integers consume fewer bytes than large ones. The value 3 in the retry_count field above occupies exactly one byte on the wire. For the high-frequency small counters and status codes typical in microservice messages, this is a consistent win. Length-delimited encoding: Strings and nested messages are encoded with an explicit byte-length prefix followed by the raw byte block. The parser reads the tag, reads the length, and copies the exact memory block directly. There is no tokenization, no string-splitting, and no key-to-field mapping via reflection. This direct memory copy approach is what makes Protobuf deserialization significantly faster than JSON parsing in practice. Benchmarks from the go_serialization_benchmarks project (available on GitHub) consistently show Protobuf outperforming standard library JSON by 4–8x in throughput on typical message shapes. Architectural Trade-Offs: When to Move and When to Wait Migrating to Protobuf is not a universal improvement. It introduces distinct operational trade-offs that teams should evaluate honestly before committing. MetricJSON over HTTP/1.1Protobuf over HTTP/2 (gRPC)Human readabilityNative — clear text in proxy logsRequires compiled schemas or tooling like grpc-curl or protoscope to inspectSchema enforcementOptional — JSON Schema is separate from the formatMandatory — enforced at build time via protoc compilationNetwork efficiencyLow — verbose string keys on every messageHigh — packed binary tag-value pairs, no key transmissionCPU utilizationHigh — heap allocation, reflection, and string parsingLow — direct memory copies and varint arithmeticDebugging overheadLow — any HTTP tool worksHigher — binary streams require schema-aware toolingSchema registry costNone — ad hoc contract managementReal — .proto files must be versioned and distributed across teams The debugging and schema-management costs deserve emphasis because they are frequently underestimated. In a JSON-based system, any engineer can inspect a live request in a proxy log or with curl. In a Protobuf system, you need the compiled schema available to decode what is on the wire. Teams that invest in a proper schema registry and standardize on tools like grpcurl absorb this cost smoothly. Teams that don't will find debugging production issues significantly harder. The Edge vs. Mesh Topology Split The most pragmatic migration approach keeps JSON at the public API boundary while adopting Protobuf exclusively for internal service-to-service traffic. The API Gateway acts as the translation layer: it terminates public-facing REST/JSON requests from browsers and mobile clients, validates the incoming payloads, and transforms them into strongly-typed Protobuf messages before routing them across the internal service mesh. Public consumers never see binary formats. Internal services get the full efficiency benefit. This topology preserves external interoperability while capturing the performance gains where they matter most, which is inside the mesh, where requests fan out across many hops. Executing a Zero-Downtime Migration The core challenge in any serialization migration is that you cannot atomically redeploy every service simultaneously. Services must continue communicating during the transition. The following phased approach handles this safely. Phase 1: Dual-Stack Services Update each internal service to accept both JSON and Protobuf requests simultaneously, using the Content-Type header to distinguish them (application/json vs. application/x-protobuf). This is the strangler fig pattern applied to serialization. No existing traffic breaks, and you can validate Protobuf behavior against live traffic without fully cutting over. Phase 2: Canary Routing Once dual-stack services are deployed, route a small percentage of internal traffic, start with 1–5%, to the Protobuf path. Monitor p99 latency, error rates, and deserialization failure metrics at the canary boundary. This is the moment where schema mismatches and field mapping errors surface, and it is far better to find them at 1% traffic than at 100%. Phase 3: Full Cutover and JSON Deprecation After the canary validates correctly over a sufficient observation window (typically one to two release cycles), shift all internal traffic to Protobuf. Maintain the JSON code path for a deprecation period to support any lagging consumers, then remove it once all services confirm clean Protobuf-only communication. Mapping JSON Structures to Proto3 When moving from a schema-less JSON environment to a typed Proto3 environment, data structures need explicit definition. Here are the most common mapping decisions. Primitive and Complex Types Numbers: Map floating-point values to double or float. Map integers to int32, int64, or uint32. If values can be negative and small (common for status codes or offsets), use sint32 or sint64, which apply ZigZag encoding to make negative varints more compact.Arrays: Represent repeated values with the repeated keyword.Maps: Use the native map<string, string> syntax. Note that map fields cannot be marked as repeated. Bootstrapping Proto Definitions From Existing Payloads When you are migrating an existing system with dozens or hundreds of active message models, writing .proto definitions by hand from legacy JSON schemas is tedious and error-prone, especially when the source payloads contain deeply nested objects, polymorphic arrays, or inconsistent field naming conventions. A practical shortcut during the early scaffolding phase is to use a JSON-to-Protobuf converter utility. You feed in a representative sample payload, and it generates a baseline .proto definition that matches the field names, infers appropriate types, and assigns initial field numbers. The output is not final. You will still need to review type choices, apply sint32/sint64 where appropriate, and add optional markers for nullable fields, but it eliminates the mechanical first pass and lets engineers focus on the decisions that actually require judgment. This is particularly useful when onboarding a new team member to the migration or when tackling a legacy service whose JSON schema was never formally documented. Handling the Absence of Native Nulls Proto3 does not have a native null state for primitive types. Unset fields default to their zero value — empty string "" for strings, 0 for integers. In systems where an unset field and a zero-value field carry different semantic meaning, this distinction matters. Two approaches address this. The first is the optional keyword, which wraps the primitive in a field-presence tracker that lets the receiver distinguish "this field was not set" from "this field was set to zero": ProtoBuf syntax = "proto3"; message PaymentRecord { string payment_id = 1; optional int32 discount_percentage = 2; // Distinguishes "no discount" from "0% discount" } The second is Google's well-known wrapper types, which provide nullable primitives at the cost of a more verbose message structure: ProtoBuf import "google/protobuf/wrappers.proto"; message ExtendedTransaction { string id = 1; google.protobuf.StringValue middle_initial = 2; // Nullable string } For most use cases, optional is the cleaner choice. Wrapper types are useful when you need to nest nullable primitives inside repeated fields or maps. Managing Schema Evolution Without Breaking Running Services In a distributed environment with independent deployment cycles, schema changes are inevitable and dangerous if handled carelessly. Protobuf addresses this through strict backward and forward compatibility rules, but only if you respect two absolute constraints. Never change field numbers. The binary parser maps incoming bytes to fields purely by tag integer. If you change a field number on a deployed message, existing services will misread the data silently and without error. Never change the wire type for an existing tag. If a field needs to change from int32 to string, you must deprecate the old tag and introduce a new field with a new field number. Beyond those hard rules, backward compatibility allows you to add new fields freely. A service that receives a message with an unknown field number will simply ignore it. This means services can be updated independently and out of order without breaking communication, which is a critical property in a rolling deployment environment. Graceful Deprecation in Practice When phasing out an existing field, mark it with the deprecated option rather than deleting it. This preserves binary compatibility for services still reading the field while alerting downstream teams through compiler warnings: ProtoBuf message UserContext { string user_id = 1; string legacy_token = 2 [deprecated = true]; // Superseded by session_hash; remove after Q3 cutover string session_hash = 3; } Do not reuse the field number after deprecation. Reserve it explicitly using the reserved keyword to prevent future developers from accidentally reusing a tag that old binary data may still contain: ProtoBuf message UserContext { reserved 2; reserved "legacy_token"; string user_id = 1; string session_hash = 3; } Concrete Implementation: Deserializing Protobuf in Go The following example shows a typical internal Go service handler receiving and deserializing a Protobuf message using the current v2 API (google.golang.org/protobuf/proto). Note: the v1 package (github.com/golang/protobuf) is archived and should not be used in new code. Go package main import ( "fmt" "log" "time" "google.golang.org/protobuf/proto" pb "path/to/generated/pb" // Pre-compiled .pb.go output from protoc ) func processPayload(rawBytes []byte) (*pb.AccountTransaction, error) { transaction := &pb.AccountTransaction{} // Unmarshal reads binary data directly into the struct without string parsing if err := proto.Unmarshal(rawBytes, transaction); err != nil { return nil, fmt.Errorf("deserialization failed: %w", err) } if transaction.GetTransactionId() == "" { return nil, fmt.Errorf("missing required field: transaction_id") } return transaction, nil } func main() { // This binary slice is the wire encoding of: // transaction_id: "tx_9988112233", account_status: "ACTIVE", retry_count: 3 // Generated via proto.Marshal on the populated AccountTransaction struct sampleBinaryPayload := []byte{ 10, 13, 116, 120, 95, 57, 57, 56, 56, 49, 49, 50, 50, 51, 51, 18, 6, 65, 67, 84, 73, 86, 69, 24, 3, } start := time.Now() tx, err := processPayload(sampleBinaryPayload) if err != nil { log.Fatalf("processing failure: %v", err) } fmt.Printf("Processed transaction %s in %v\n", tx.GetTransactionId(), time.Since(start)) } The key difference from JSON unmarshaling is in what proto.Unmarshal does not do: it does not tokenize strings, does not map keys via reflection, and does not allocate intermediate string representations. It reads the tag, determines the field type from the compiled schema, and copies raw bytes directly to the target struct field. At high throughput, that distinction in allocation behavior is what drives the difference in GC pressure and tail latency. What This Migration Actually Solves, and What It Does Not Protobuf is not a solution to every distributed systems problem. It will not fix poorly designed service boundaries, reduce round trips caused by chatty interfaces, or compensate for network topology problems. What it specifically addresses is the serialization and deserialization overhead on hot paths where internal services are exchanging high volumes of structured messages. The teams that see the clearest wins are those where profiling has confirmed that serialization CPU time is a meaningful contributor to request latency, and where payload sizes have made bandwidth a real infrastructure cost. If your p99 latency problems trace to database queries, downstream API calls, or lock contention, the Protobuf migration will have minimal impact on those numbers. Start by profiling your highest-traffic internal endpoints. Measure serialization time as a fraction of total request time. Measure payload sizes across a representative sample of production traffic. If the data shows serialization is a genuine bottleneck, the migration is well-justified. If it is not, the operational investment in schema management and tooling upgrades may not pay off on the timeline you need. For the services where it does make sense, the gains are real and durable. Lower CPU utilization, reduced GC pressure, smaller payloads across every internal hop, and strongly typed contracts enforced at build time; these compound over time as traffic grows. Summary The path from JSON to Protobuf is not about chasing a trend. It is a deliberate architectural decision to eliminate serialization overhead on hot internal paths by replacing text parsing with direct binary memory operations. The practical steps are straightforward: audit your highest-traffic internal endpoints, define your .proto schemas with careful attention to field numbering and null semantics, deploy dual-stack services to enable a phased cutover, and establish tooling for schema versioning before your team's first production deployment. The operational costs are real but manageable. Binary streams require schema-aware debugging tools, .proto files need disciplined version management, and the reserved keyword must become part of your deprecation workflow. Teams that treat schema governance as a first-class concern alongside their code absorb these costs smoothly. For distributed systems where internal traffic volume makes serialization overhead measurable, the migration consistently delivers: lower tail latency, reduced bandwidth spend, and contracts that fail loudly at compile time rather than silently at runtime.

It’s time to meet another member of the DZone community! Abhishek Sharma is a newer face at DZone, but he has already made a great impact. I caught up with him to learn more about his journey into tech and what keeps him curious both in and outside of work. What first got you interested in technology? "What first drew me to technology was seeing how it could solve real business problems. Early in my career, I realized that technology becomes much more interesting when you understand what is happening behind the system — how a business operates, where people struggle, and how technology can simplify that experience. That curiosity stayed with me as I moved from working with enterprise applications into CRM, customer experience, field service, cloud transformation, and enterprise architecture. Over the years, the technologies have changed significantly, but what continues to interest me is the same question: How can we use technology to make a complex business process work better for the people who actually depend on it?" What’s one tool you couldn’t work without? "I would probably say a good architecture diagram — or even a simple whiteboard. My work often involves bringing together business processes, enterprise applications, integrations, data, AI, and operational teams. When a problem becomes complicated, visualizing it usually makes the conversation much easier. Whether I am discussing CRM, field service, inventory, ERP, AI, or integration architecture, putting the end-to-end flow in front of everyone helps people see dependencies that may otherwise be missed. I have learned that sometimes a well-designed diagram can resolve in twenty minutes what several meetings could not." What’s your favorite way to keep your technical skills current? "For me, the best way to stay current is to combine structured learning with practical application. I continue to pursue certifications and explore new capabilities, particularly around Oracle Cloud, Field Service, AI, agentic AI, automation, and enterprise architecture, but I do not like learning technology only at a theoretical level. I learn much more by asking how an emerging technology would actually work in a real enterprise environment. Writing technical articles also helps because it forces me to organize my thinking and challenge my own assumptions. Judging technology awards, engaging with professional organizations, reading industry research, and learning from other architects and practitioners give me perspectives outside my immediate projects as well. Technology changes too quickly to ever say, “I know enough.” Continuous learning has simply become part of the profession for me." In your free time, what do you like to do? "I enjoy hiking and spending time with my kids, especially playing games with them. What I enjoy most about spending time with my kids is seeing the world through their eyes. They often approach a game or a problem with a completely different perspective, and it’s a great reminder that sometimes the best ideas come from looking at familiar things in unfamiliar ways." Hiking sounds wonderful! Do you have any pictures to share? "I have attached a picture of me clicked during one of the Hiking trails near Cuyahoga Falls in Ohio. Hiking is one of my favourite ways to step away from technology and spend time outdoors with his family. For me, it provides a chance to slow down, recharge, and enjoy time away from the demands of work." Check out more of Abhishek's content here.

Playwright is a widely used open-source test automation framework developed by Microsoft. It allows developers and test automation engineers to reliably automate web applications across multiple browsers and platforms. Playwright supports several popular programming languages, such as JavaScript, TypeScript, Java, C#, and Python. One of its standout features is built-in API automation testing, which gives it a strong advantage over many traditional web automation frameworks. In this tutorial, we’ll explore how to use Playwright with TypeScript and learn how to automate GET API requests. Installing Playwright With TypeScript The first step is to install and set up Playwright with TypeScript. Let’s create a new folder and run the following command by navigating to the newly created folder: Plain Text npm init playwright@latest After running the above command, make sure you select “TypeScript” as the programming language. Next, select the appropriate options for the other questions asked by the Playwright setup and install Playwright and its dependencies. Application Under Test We’ll be using free, publicly available RESTful e-commerce APIs from a demo e-commerce application hosted on GitHub. The project can be run locally using either Node.js or Docker and provides several order management APIs, including creating, updating, retrieving, and deleting orders. How to Test GET API Requests With Playwright TypeScript Playwright provides a request API that lets us create and manage HTTP request contexts. Let’s learn about sending GET requests step-by-step with different options: Send a GET API Request and Verify the Status Code Let’s perform a simple test by sending a GET API request and verifying that a 200 status code is returned in the response. TypeScript import { test, expect } from "@playwright/test"; test("Get Order details API test with status code check", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, }); expect(response.status()).toBe(200); }); Code Walkthrough This test sends a GET request to the /getOrder API with a user_id parameter using Playwright’s request context. It verifies that the API responds successfully by checking that the status code returned is 200. The following are additional details about this test: test(…): The test(…) defines a Playwright test case. The string “Get Order details API test with status code check” is the name of the test and will be shown in the Playwright report.async ({ request }): It uses Playwright’s built-in request fixture, which injects an APIRequestContext and allows us to make HTTP calls.Sending a GET request: The following line sends an HTTP GET request to the /getOrder/ endpoint. TypeScript const response = await request.get("http://localhost:3004/getOrder/", { The await keyword pauses execution until the API responds. Finally, the result is stored in the response variable, which is an APIResponse object. Params: The following line adds a query parameter “user_id” to the GET request. TypeScript params: { user_id: "1", }, expect statement: The response.status() retrieves the HTTP status code returned by the API, and expect(…).toBe(200) asserts that the API responded successfully with HTTP 200 OK. Similarly, we can perform the assertions for a status code other than 200. In the code below, the value for the “id” parameter is updated to “2”, for which no records exist in the system. TypeScript test("Get Order details API test with status code 404", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { id: 2, }, }); expect(response.status()).toBe(404); }); The expectation is that it should return status code 404. The expect(...) statement performs the required status code check. Send a GET API Request With Multiple Parameters There are situations where we need to provide multiple parameters in the GET request to filter and fetch the required records. Using Playwright TypeScript, multiple parameters can be supplied while sending a GET request, as shown below: TypeScript test("Get Order details API test with multiple params", async ({ request }) => { const params = { id: 1, user_id: "1", product_id: "79", }; const response = await request.get("http://localhost:3004/getOrder/", { params, }); expect(response.status()).toBe(200); }); This test defines multiple query parameters (id, user_id, and product_id) in a single params object and sends them with a GET API request. Playwright automatically appends these parameters to the request URL. Send a GET API Request With Headers Headers play an important role in retrieving data from the server. They can be supplied in the GET request as shown below: TypeScript test("Get Order details API test with headers", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { id: 1, user_id: "1", }, headers: { ContentType: "application/json", }, }); expect(response.status()).toBe(200); }); This test sends a GET request with custom HTTP headers along with query parameters, where the headers option is used to specify that the request content type is JSON. Similarly, other headers such as “Authorization”, “Accept”, “User-Agent”, etc. can also be supplied. Send a GET API Request With a Timeout Option Playwright provides the timeout option that can be passed to the request.get() method for setting a timeout to limit how long to wait for the response. TypeScript test("Get order details API test with timeout", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: 1, }, headers: { ContentType: "application/json", }, timeout: 300, }); expect(response.status()).toBe(200); }); If the API does not respond within the given timeout, Playwright fails the request and throws a timeout error. It helps prevent tests from hanging and makes failures faster and more predictable, especially for slow or unstable APIs. Send a GET API Request With the failOnStatusCode Option The failOnStatusCode option tells Playwright to automatically fail the request if the API responds with a non-2xx status code (such as 400, 404, 500, etc). TypeScript test("Get order details API test with fail on status code", async ({ request, }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, headers: { ContentType: "application/json", }, failOnStatusCode: true, }); }); Using this option, we can get rid of performing the checks using response.status() as Playwright throws an error immediately if the API does not respond with a 2xx status code. The failOnStatusCode option is useful when a request must succeed for the test to continue. For example, if we need to validate the response data, we must use this option to ensure that the API responds with a 2xx status code before proceeding with deeper response validation. Test Execution Let's execute all the tests that we discussed and also check the built-in report provided by Playwright. To run the tests, execute the following command from the terminal: Plain Text npx playwright test After the test execution is complete, the built-in Playwright report can be generated using the following command: Plain Text npx playwright show-report The report shows details of the test run, including test names, time taken, the browser agent used, and the number of tests executed, along with their pass/fail status. Watch the step-by-step YouTube tutorial on how to test GET API requests with Playwright TypeScript. Summary Testing GET API requests with Playwright using TypeScript allows you to easily send requests with query parameters and custom headers while keeping your tests clean and readable. Playwright also provides options such as timeout to control request duration and failOnStatusCode to automatically fail tests on non-successful responses. Together, these features help test the GET API requests efficiently.

By Faisal Khatri CORE

Fifteen years in, and the conversation I have most often with security leads still starts the same way: how's your perimeter, how's your endpoint coverage, how's your SOC staffed? Almost nobody opens with "how's your pipeline." That's the gap I want to talk about, because 2025 was the year the gap turned into a crater. Here's the number that should reorder every security roadmap for 2026: major DevOps platforms — GitHub, GitLab, Azure DevOps, and Atlassian's Jira and Bitbucket — patched 236 vulnerabilities in 2025, according to GitProtect's DevOps Threats Unwrapped report. Of those, 59% were rated high or critical: 14 critical, 126 high, 75 medium, 21 low. The trend line is worse than the total. Critical flaws jumped from 4 in the first half of the year to 10 in the second. High-severity findings climbed 55%, from 39 to 87 over the same stretch. November 2025 alone produced 36 patched vulnerabilities — 15% of the entire year's total in one month. These aren't obscure internal tools. GitHub alone hosts more than 180 million developers across 630 million repositories. When the platforms holding that much code accelerate their vulnerability disclosures quarter over quarter, that's not noise. That's a trend with a direction (DevOps.com, SecurityBrief). "You Hack the Runner, and You're in the Whole Building With a Master Key" I want to be careful here and use a real source, because this argument gets thrown around a lot without anyone actually backing it up. Paweł Budzan, a technology consultant and AI and cybersecurity architect at Xopero, put the stakes in terms I haven't heard bettered: a compromised CI/CD pipeline hands an attacker the repo, the cloud, the production secrets, and the deployment path all at once. Hack the runner, he said, and you're not in one room — you're in the whole building with a master key to every door. His point about "shift-left" culture stung a little because I've made the same mistake myself: plenty of teams scan the code, call it a day, and never apply the same scrutiny to the infrastructure that actually moves that code into production (GitProtect/Xopero). Budzan's list of the ten most commonly overlooked CI/CD vulnerabilities reads like a checklist written by someone who's cleaned up after every one of them: secrets echoed into build logs and never scrubbed; runners granted full cluster-admin rights because restricting them "might break the build"; a single shared runner handling both untrusted external pull requests and production deployments; blind trust in third-party GitHub Actions with a few stars and no real vetting; long-lived service tokens nobody rotates because rotation is scary; unprotected workflow YAML files that get far less code-review scrutiny than application logic. None of these are exotic. That's exactly the problem. When AI Turns a Script Kiddie Into a Supply-Chain Threat The part of Budzan's analysis that actually changed how I think about this: the barrier to entry for a serious pipeline attack has, in his words, dropped to the level of writing prompts in English. Malicious large language models — he named WormGPT and FraudGPT specifically, tools sold on dark-web forums and Telegram channels for a monthly fee — are trained specifically for offensive use, with none of the guardrails a mainstream model would apply. An attacker doesn't need deep AWS or Git expertise anymore. They can feed a workflow YAML file into one of these tools and ask it to locate secrets or draft a plausible-looking pull request. What comes back is a clean, credible "fix" that sails through code review, and when the pipeline runs, the token leaks straight out. Budzan's own estimate of the human-versus-sophistication split surprised me: from his practice, it's roughly 80% human error and 20% advanced attack — a developer under sprint pressure who leaves a token in a config file, tells themselves they'll fix it after the weekend, and never does (GitProtect/Xopero). The real-world version of that pattern already happened. In March 2025, attackers compromised the popular tj-actions/changed-files GitHub Action — retroactively rewriting version tags to point at a malicious commit — and the poisoned action ended up exposed in more than 23,000 repositories before it was caught and patched. It didn't require breaking any cryptography. It required trust in a dependency nobody was individually vetting (GitHub Advisory Database). The CI/CD Tools Themselves Are Now the Target, Not Just the Delivery Mechanism Then there's the newer wrinkle: the AI tooling that's increasingly wired directly into these pipelines is itself shipping critical flaws. In April 2026, researchers at Novee Security disclosed a maximum-severity, CVSS 10.0 remote code execution vulnerability in Google's Gemini CLI and its companion run-gemini-cli GitHub Action — a flaw that let an unprivileged external attacker force their own malicious content to load as the tool's configuration, effectively turning an AI coding assistant embedded in a CI/CD workflow into a remote-execution foothold. Google assigned it the highest score the scale allows. This is the pattern Budzan's framework predicts almost exactly: the pipeline as master key, an AI tool as the unguarded runner, and a single crafted input as the way in (Novee Security). The Uncomfortable Final Word: No Defense Is 100%, So Plan For the Day It Fails What I respect about Budzan's take is that he doesn't oversell prevention. Backup and disaster recovery, he argues, are the actual last line of defense — and anyone who claims a tool stops 100% of attacks is selling you something, full stop. His specific standard for what counts as a real backup is worth repeating exactly because it's so unglamorous: isolated, offline, in a separate cloud tenant and a separate account — not a separate folder in the same S3 bucket, and definitely not sitting in the same AWS account as production. If the same compromised credentials that let an attacker into your pipeline can also reach your backups, you don't have a backup. You have a second copy of the crime scene (GitProtect/Xopero). Where This Leaves Security Leaders Every thread here points the same direction: CI/CD pipelines have quietly become as valuable a target as production itself, and in most organizations they're guarded with a fraction of the rigor. The 236 patched vulnerabilities, the accelerating severity curve through the back half of 2025, the tj-actions compromise, the Gemini CLI RCE, and Budzan's own field experience all describe the same failure mode from different angles: trust extended to a pipeline, a runner, a third-party action, or an AI assistant, with nobody watching that trust closely enough. I don't think the fix is more scanning tools bolted onto the same workflow. It's treating the pipeline itself — the runners, the tokens, the workflow files, the AI assistants wired into it — with the same access control, isolation, and adversarial testing you'd apply to a production database. And it's accepting, the way Budzan does, that prevention will eventually fail, so the backup sitting behind it needs to be somewhere the attacker who got in through the front door can't also reach. Most teams I talk to still don't have that. The threat landscape isn't going to wait for them to build it. Sources are linked inline throughout. Reporting and analysis are current as of July 2026.

By Igboanugo David Ugochukwu CORE

Snowflake has become a go-to platform for storing and querying operational data at scale. SQL is excellent at filtering rows, joining tables, and aggregating numbers. But there's a class of questions where SQL starts to struggle: questions about connections. Which machines in a production line depend on this one? If this component fails, what else goes down with it? Which assets play equivalent structural roles across parallel workflows? These are fundamentally questions about relationships, and answering them in SQL requires increasingly complex recursive queries as the number of hops grows. Graph analytics is a natural complement here. Rather than replacing SQL, it adds a new lens on data you already own. In this article, we'll see how to use the Neo4j Graph Analytics Native App, available from the Snowflake Marketplace, to run graph algorithms directly on Snowflake tables — no data movement, no separate infrastructure, no new data store to maintain. The full source code is available on GitHub. The Scenario We'll work with a manufacturing plant dataset: 20 machines (Cutters, Welders, Presses, Assemblers, and Painters) connected by directed material flow relationships. Each machine has a risk level (low, medium, or high), and each relationship carries a throughput rate. This is representative of the kind of operational data that already exists in Snowflake for real systems — asset registers, process flows, supply chain graphs. The questions we ask of it apply equally to those domains. Setup The Neo4j Graph Analytics app is installed from the Snowflake Marketplace — just search for "Neo4j." Once installed, we'll create a database, load the demo data, and configure the permissions the app needs to read from and write to our tables. The data lives in two tables: nodes (one row per machine) and rels (one row per material flow connection). Graph algorithms need a simplified view of these — just node IDs and source/target pairs — so we create two projection-ready tables: Python # Node view - just the IDs, which is what graph projections need session.sql(""" CREATE OR REPLACE TABLE ga_demo.public.nodes_vw AS SELECT machine_id AS nodeId FROM ga_demo.public.nodes """).collect() # Relationship view - aggregate to ensure one weight per pair session.sql(""" CREATE OR REPLACE TABLE ga_demo.public.rels_vw AS SELECT src_machine_id AS sourceNodeId, dst_machine_id AS targetNodeId, CAST(SUM(throughput_rate) AS FLOAT) AS total_amount FROM ga_demo.public.rels GROUP BY src_machine_id, dst_machine_id """).collect() Thinking in Graphs Before running algorithms, it's worth establishing a shared vocabulary. A graph is made of nodes (entities) and relationships (connections between them). Both can carry properties. In our plant, each machine is a node — its machine_type and risk_level are properties on that node. Each material flow connection is a relationship — its throughput_rate is a property on that relationship. The data are already in Snowflake. A graph is not a separate thing you import data into. It's a lens on data you already own. Every algorithm call in Neo4j Graph Analytics includes a project block that tells the app which Snowflake tables to use as nodes and which to use as relationships. The app reads those tables, builds a temporary in-memory graph structure, runs the algorithm, writes results back to a Snowflake table we specify, and then discards the in-memory structure. Our data never leaves Snowflake. We can visualize the plant graph before running any algorithms to get a sense of its structure. Figure 1. Manufacturing Plant Graph A few things are immediately visible: one node appears to receive connections from many others, and one node seems to sit between otherwise separate sections of the plant. The algorithms that follow will confirm these observations numerically. Connectivity Analysis: Weakly Connected Components Our first question is foundational: is this plant one integrated system, or does it split into isolated subsystems? Weakly Connected Components (WCC) treat the graph as undirected — it ignores the direction of material flow and asks simply: can every machine reach every other machine through some path? The output assigns each machine a component ID. Multiple component IDs would indicate isolated sub-plants. Python session.sql(""" CALL neo4j_graph_analytics.graph.wcc('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': {}, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_wcc' }] }) """).collect() The results show a single component containing all 20 machines — the plant operates as one integrated network. This is a useful baseline: it tells us there are no isolated subsystems that might be invisible to centralized monitoring. Criticality Analysis: PageRank and Betweenness Centrality Knowing the plant is connected, we can ask: which machines are most critical? We use two algorithms that measure criticality in different ways. A machine can be critical for one reason but not the other, and the distinction has real operational implications. PageRank: Flow Importance PageRank asks which machines receive material from many well-connected upstream machines. A high PageRank score means a machine is a destination for flow from important sources. If it slows down, the backlog ripples upstream. Python session.sql(""" CALL neo4j_graph_analytics.graph.page_rank('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'score' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_pagerank', 'nodeProperty': 'score' }] }) """).collect() Machine 20 comes out on top — it sits at the confluence of multiple upstream chains, the assembly hub where material from across the plant converges. Figure 2. PageRank Visualization Betweenness Centrality: Structural Importance Betweenness asks a different question: which machines appear most often on the shortest path between other machines? A high Betweenness score means a machine is a structural bridge. It may not handle the most flow, but its position connects otherwise separate parts of the plant. If it goes offline, it disconnects or lengthens paths across the network. Python session.sql(""" CALL neo4j_graph_analytics.graph.betweenness('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'score' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_betweenness', 'nodeProperty': 'score' }] }) """).collect() Machine 3 has the highest Betweenness score — despite having a much lower PageRank than Machine 20. It's not the busiest machine; it's the one whose failure would do the most structural damage. Figure 3. Betweenness Centrality Heatmap This is the key insight from running both algorithms: PageRank and Betweenness reveal different kinds of importance. A maintenance plan that uses only one of them is missing half the picture. Structural Similarity: FastRP and KNN So far we've identified individual critical machines. This section asks a different question: which machines play the same structural role in the workflow, even if they're different types? Machines with structurally equivalent positions can share maintenance windows, act as backups for each other, or be treated as a unit for risk modeling — even if they look different on paper. We use two algorithms in sequence. Fast Random Projection (FastRP) FastRP generates a compact embedding vector for each machine by sampling the graph structure around it. Two machines with similar upstream and downstream neighbors will end up with similar embedding vectors, regardless of their type or risk level. We use 16 dimensions — a good balance for a 20-node graph. Python session.sql(""" CALL neo4j_graph_analytics.graph.fast_rp('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'embedding', 'embeddingDimension': 16 }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_fastrp', 'nodeProperty': 'embedding' }] }) """).collect() K-Nearest Neighbor (KNN) KNN takes the embeddings and finds, for each machine, its most structurally similar peer. Similarity is measured using cosine similarity of the embedding vectors — a score of 1.0 means identical structural position, 0.0 means completely different. Note that KNN operates on node properties rather than graph edges, so its projection block contains no relationship table — the one exception to the pattern seen in the other algorithm calls. Figure 4. KNN Structural Similarity Matrix The results show high-similarity pairs between machines of different types. This is expected: FastRP captures structural position in the graph, not machine attributes. Two machines with similar upstream and downstream neighbors will have similar embeddings regardless of their type, risk level, or throughput rate. Failure Simulation Static risk analysis tells us which machines are currently important. We can turn that into a dynamic tool by asking: what actually happens to the rest of the plant when Machine 3 goes offline? We simulate the failure by creating filtered views that exclude Machine 3 and all its connections, then re-run PageRank and Betweenness on the degraded graph. Normalization matters here: raw scores shrink after failure because the graph is smaller. We divide each score by the sum of all scores in that run so we're comparing relative importance within each graph, not absolute values. Python session.sql(f""" CREATE OR REPLACE VIEW ga_demo.public.nodes_failure_vw AS SELECT machine_id AS nodeId FROM ga_demo.public.nodes WHERE machine_id != {EXCLUDED} """).collect() session.sql(f""" CREATE OR REPLACE VIEW ga_demo.public.rels_failure_vw AS SELECT src_machine_id AS sourceNodeId, dst_machine_id AS targetNodeId, CAST(SUM(throughput_rate) AS FLOAT) AS total_amount FROM ga_demo.public.rels WHERE src_machine_id != {EXCLUDED} AND dst_machine_id != {EXCLUDED} GROUP BY src_machine_id, dst_machine_id """).collect() Figure 5. Betweenness Delta Bar Chart The key finding: machines that were not flagged as high risk in the baseline analysis gain significant Betweenness importance after Machine 3's failure. The network reroutes through alternative paths, promoting machines that were structurally insignificant in the baseline into critical bridge positions. Static risk labels don't capture this — graph analysis does. The notebook is designed to support experimentation: change the EXCLUDED variable to any machine ID and re-run the section to see how the network responds to a different failure. Community Detection: Louvain The previous sections analyzed individual machines. Louvain community detection asks: does the plant naturally organize itself into clusters? Louvain finds groups of machines that are more densely connected to each other than to the rest of the network. These communities often correspond to real operational sub-units — parallel production lines, shared workflow stages, or tightly coupled machine groups. Python session.sql(""" CALL neo4j_graph_analytics.graph.louvain('CPU_X64_XS', { 'project': { 'defaultTablePrefix': 'ga_demo.public', 'nodeTables': ['nodes_vw'], 'relationshipTables': { 'rels_vw': { 'sourceTable': 'nodes_vw', 'targetTable': 'nodes_vw' } } }, 'compute': { 'mutateProperty': 'community' }, 'write': [{ 'nodeLabel': 'nodes_vw', 'outputTable': 'ga_demo.public.nodes_louvain', 'nodeProperty': 'community' }] }) """).collect() Figure 6. Louvain Community Detection Joining the community results back to the risk levels reveals that the smaller community has a disproportionate concentration of high-risk machines relative to its size. This also explains the failure simulation results: Machine 3 sits in this community and acts as its main bridge to the rest of the plant. Community detection connects the structural analysis back to operational risk in a way that neither algorithm produces on its own. Bringing It All Together The final step joins all four algorithm outputs into a single risk summary table: Python risk_summary = session.sql(""" SELECT n.machine_id, n.machine_type, n.risk_level, ROUND(p.score, 4) AS pagerank_score, ROUND(b.score, 4) AS betweenness_score, l.community FROM ga_demo.public.nodes n JOIN ga_demo.public.nodes_pagerank p ON n.machine_id = p.nodeid JOIN ga_demo.public.nodes_betweenness b ON n.machine_id = b.nodeid JOIN ga_demo.public.nodes_louvain l ON n.machine_id = l.nodeid ORDER BY pagerank_score DESC """).to_pandas() This table combines flow importance, structural importance, and community membership into a single view — one that would be difficult to produce from SQL alone and impossible without running the underlying graph algorithms. Summary SQL and graph analytics aren't competing approaches — they're complementary ones. Snowflake handles what it does well: storing, filtering, and aggregating operational data at scale. Neo4j Graph Analytics, running as a Native App inside Snowflake, adds a layer of analysis that SQL alone can't easily provide: understanding how entities relate to each other, which ones are structurally critical, and how the network behaves under failure conditions. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Hadera, Israel, September 8th, 2026, TechnologyWire This article was provided by TechnologyWire and does not represent the editorial content of DZone. RavenDB, a NoSQL document database used by more than 12,000 customers, announced today the launch of its new product, Quill, a context layer for SQL databases that makes them ready for production AI agents without migrating the system of record or architecting a custom AI stack. With AI becoming a board-level mandate, CTOs and VPs of engineering are under pressure to ship AI capabilities fast. But for organizations whose mission-critical data sits in legacy SQL systems, built years before embeddings or agents existed, AI can’t access their data. Modernizing or replacing the systems is expensive, risky, and time-consuming. By the time the system is updated, nobody remembers what the project was supposed to achieve or how it measured ROI. Recently, a Gartner survey of infrastructure and operations leaders found that one in five AI initiatives fail, and only 28% report a positive ROI, which is linked to how well the technology is integrated, governed, and aligned with operational needs, not to the sophistication of the model. As AI becomes the industry standard, organizations have been left without a clear path to deliver until now. "Anyone can stand up an AI demo in an afternoon, but getting that demo into production with data pipelines, semantic search, security, governance, all the plumbing a small proof of concept doesn't need until it has to run at scale, is the hard part," said Oren Eini, founder and CEO of RavenDB. "Quill exists because we'd rather hand teams that plumbing already assembled than watch them rebuild the same project after project. You get access to the live data you need, decide the scope on day one, and change it as you go, instead of building everything from scratch." Quill connects directly to an organization's existing SQL database and adds a context layer on top, making it possible to launch production-ready agents in weeks rather than the 18 to 24 months of a typical in-house build. The source system stays exactly where it is and remains authoritative, and the full AI stack- search, retrieval, and agents that can answer questions- is included. Agents built on Quill support web chat, WhatsApp, Telegram, Slack, and Discord out of the box. "With Quill, the plumbing was already there, so we spent our time building the actual feature," said Hagay Albo, CEO at Albos Technologies and Holdings, an early adopter of Quill. By default, Quill is governed, sitting between the AI and the source system, and it is built on the assumption that the model itself cannot be trusted with unrestricted access, so organizations decide exactly what an agent can and cannot see, independent of the source database's own permissions. In a healthcare setting, for example, an agent can answer a patient's question about an upcoming appointment, while prescription data is never part of the dataset it can query. What is usually a custom security project becomes a configuration choice. Quill is also model-agnostic, so teams can use any AI model, switch providers, or run entirely on their own hardware. Quill is now available for organizations running PostgreSQL, SQL Server, or MySQL, with more databases to be supported in the future, and can be deployed in the cloud or on-premises to meet data-residency or regulatory requirements. To start using Quill today, visit: https://ravendb.net/quill About RavenDB: RavenDB is a hybrid NoSQL document database built for modern application development. Used by more than 12,000 customers across 50 industries, RavenDB helps teams move faster with seamless data management across cloud, on-prem, and edge environments. With full-text search, automatic indexes, and an easy-to-use studio for monitoring and administration, RavenDB is the database developers love and enterprises trust. Learn more at www.ravendb.net

Modern database environments rarely run a single type of workload. Most production systems handle both transactional operations and analytical queries simultaneously. These mixed workloads, often referred to as hybrid workloads, place significant pressure on traditional database indexing and storage strategies. In such environments, disk-based indexes can become a performance bottleneck. When transactional and analytical queries compete for disk I/O, it often results in increased latency, reduced throughput, and inconsistent query performance. To address these challenges, SQL Server leverages memory-optimized tables and indexes as part of its In-Memory OLTP capabilities. These features reduce reliance on disk I/O by enabling data and index access directly from memory, while still maintaining durability through logging and checkpoint mechanisms. This article explores how memory-optimized indexing works and demonstrates how it can significantly improve performance in real-world hybrid workload scenarios. Core Characteristics Mandatory inclusion: Every memory-optimized table must have at least one index, as they serve as the "entry points" for row access.Purely in-memory: Indexes are rebuilt entirely from scratch during database recovery based on their definitions and the data loaded into memory.Non-persistent: Unlike traditional indexes, changes to these indexes are not written to the transaction log, reducing I/O overhead.Fragmentation-free: These structures do not suffer from traditional page fragmentation, eliminating the need for regular REORGANIZE or REBUILD operations. Index TypeBest Use CaseBehaviorHash IndexEquality SearchesUses an array of buckets; highly efficient for point lookups (e.g., WHERE ID = 5).Nonclustered IndexRange QueriesUses a lock-free B-tree structure (Bw-tree); ideal for range scans and sorted results (e.g., WHERE Price > 100). The Challenge With Traditional Indexing Traditionally, database indexes are stored on disk to ensure durability. While this design protects data, it introduces a major limitation: disk I/O latency. In environments with heavy workloads, disk access becomes a bottleneck. This is particularly noticeable when: Large analytical queries scan index rangesTransactional queries require fast point lookupsMany concurrent users access the system When both workloads run together, index operations often compete for disk resources, resulting in slower queries and higher latency. Introducing Memory-First Indexes Memory-First Indexes in SQL Server 2025 take a different approach. Instead of relying primarily on disk-based indexes, the system prioritizes in-memory index access for frequently used data while maintaining a synchronized copy on disk for durability. The key idea is simple: Hot data (frequently accessed index ranges) is kept in memory.Cold data remains on disk.Changes made in memory are synchronized with disk replicas in the background. This approach allows SQL Server to serve many queries directly from memory while still maintaining persistence. The feature also includes monitoring mechanisms that track query patterns. When the system detects frequently accessed index partitions, it moves them into memory automatically. Less frequently accessed portions are pushed back to disk to conserve memory resources. The result is faster query execution without requiring manual tuning from database administrators. Real-World Example: Retail E-Commerce Database To understand the benefits, consider a retail company running an e-commerce platform. The company stores millions of products in a table with the following structure: ProductID – unique identifierProductCategory – category of the productPrice – product priceStockQuantity – available inventory The application runs two types of queries. Transactional Query This query checks stock availability for a specific product. SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; Analytical Query This query calculates aggregated metrics by product category. SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; In a traditional setup, both queries rely on disk-based indexes. When concurrency increases, disk access becomes saturated, and query performance suffers. With Memory-First Indexes, the most frequently used index ranges, such as ProductID and ProductCategory, are loaded into memory, allowing much faster lookups. Testing the Feature To evaluate the impact of Memory-First Indexes, we can simulate a large dataset and compare query performance before and after enabling the feature. Step 1: Create the Table SQL CREATE TABLE Products ( ProductID INT PRIMARY KEY, ProductCategory NVARCHAR(50), Price DECIMAL(10,2), StockQuantity INT ); Step 2: Populate Test Data The following script generates a large dataset for testing. SQL INSERT INTO Products (ProductID, ProductCategory, Price, StockQuantity) SELECT TOP 50000000 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS ProductID, CASE WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 1 THEN 'Electronics' WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 2 THEN 'Clothing' ELSE 'Home Appliances' END AS ProductCategory, ABS(CHECKSUM(NEWID()) % 1000) + 1.00 AS Price, ABS(CHECKSUM(NEWID()) % 5000) + 1 AS StockQuantity FROM sys.all_objects a CROSS JOIN sys.all_objects b; Step 3: Create Traditional Indexes SQL CREATE INDEX IX_Products_ProductID ON Products (ProductID); CREATE INDEX IX_Products_Category ON Products (ProductCategory); At this stage, run the transactional and analytical queries and capture baseline metrics using Query Store or dynamic management views. Step 4: Enable Memory-First Indexes Next, recreate the indexes with Memory-First enabled. SQL DROP INDEX IX_Products_ProductID ON Products; CREATE INDEX IX_Products_ProductID ON Products (ProductID) WITH (MEMORY_FIRST = ON); DROP INDEX IX_Products_Category ON Products; CREATE INDEX IX_Products_Category ON Products (ProductCategory) WITH (MEMORY_FIRST = ON); Step 5: Execute Test Queries SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; MS SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; Record execution time, CPU usage, and disk activity again. Observed Performance Improvements The results typically show noticeable performance gains. For example: Transactional queries Before: ~50 msAfter: ~15 ms Analytical queries Execution time reduced by about 50% System metrics also reveal additional improvements: Disk I/O reduced by more than 70%Memory usage increased only moderatelyCPU utilization became more stable during peak workloads These improvements occur because queries are able to retrieve indexed data directly from memory rather than waiting for disk operations. Why This Matters for Modern Workloads Hybrid workloads are becoming the norm across many industries, including retail, finance, and IoT platforms. Systems must support both real-time transactions and large analytical queries without sacrificing performance. Memory-First Indexes help address this challenge by: Reducing disk I/O bottlenecksImproving response time for critical queriesAutomatically adapting to changing workload patternsMaintaining durability with synchronized disk replicas Final Thoughts Memory-First Indexes represent an important improvement in SQL Server 2025’s indexing architecture. By prioritizing in-memory access for frequently used data, SQL Server can deliver significantly faster query performance while still preserving data durability. For organizations running mixed transactional and analytical workloads, this feature can reduce latency, improve system stability, and make better use of available hardware resources. As hybrid workloads continue to grow, features like Memory-First Indexing will play a key role in helping database platforms keep up with modern application demands.

By arvind toorpu CORE