Data Engineering
Kafka Rebalancing on Amazon MSK: Moving Beyond Stop-the-World Pauses
Kafka consumer recovery is moving from a global pause model toward incremental, broker-directed coordination.
Insights
Latest analysis
Explore changes in data engineering, AI, analytics and cloud platforms, with practical implications for your team.
Latest article: 3 October 2026
Search and filter insights
Quick categories
Article index
Showing 9 of 26 insights
Modern Data Stack
Data Platforms Are Bringing Operability Closer to the Workload
The latest changes across the data-platform landscape are not simply adding more features. They are bringing development, security, resilience and governance closer to the workloads that data teams operate every day.
Data Engineering
Delivery Hero’s Flink Rebuild: From Hourly Measurement to Near Real Time
Streaming modernisation can improve both decision speed and operating cost when it replaces repeated batch processing with a carefully designed stateful pipeline.
Data Engineering
Moeve’s Shared dbt Launcher: Standardising Execution Without Centralising Every Pipeline
Data teams can standardise how transformations run while keeping orchestration and domain-owned data loosely coupled.
AI for Data Teams
SageMaker Catalog: Governed Unstructured Data Becomes Queryable by Conversation
Enterprise unstructured-data workflows are moving toward the same catalogue, ownership and approval patterns used for governed analytical data.
Data Governance
SageMaker Tooling Blueprints: Permissions Boundaries Reach Provisioned Roles
Governed data and AI workspaces are beginning to inherit enterprise permission limits when they are created.
Data Engineering
Spark Connect on Amazon EMR: Local Development Meets Remote-Scale Execution
Distributed data development is separating the developer interface from the cluster where execution happens.
BI & Analytics
BigQuery augmented analytics: investigating metrics in SQL
Analytical workflows that previously required separate statistical notebooks or specialised tooling are increasingly becoming callable warehouse functions.
Modern Data Stack
BigQuery Graph: connected data in the warehouse
Graph workloads no longer necessarily require organisations to move warehouse data into a separate graph database.
Data Engineering
Databricks Auto CDF: deriving changes at query time
Change-data processing is becoming less dependent on configuring every table correctly before changes occur.
BI & Analytics
Databricks Genie: more context needs more curation
Conversational analytics products are beginning to encounter the same scale, context and governance questions as conventional BI.
Data Engineering
Dataflow Pause/Resume: planning for batch recovery
Resilience features are becoming increasingly important as data pipelines take on expensive AI-processing workloads.
Data Engineering
dbt State: change-aware pipelines
Data transformation is starting to move from run everything on a schedule toward run what actually needs to change.
BI & Analytics
Fabric Warehouse Monitor: query observability in practice
Warehouse monitoring is becoming a first-class part of the analytical development experience rather than something teams assemble afterwards.
Data Governance
Iceberg Read Restrictions: portable policy needs trusted engines
Portable table access needs a policy contract between the catalog and trusted query engines.
AI for Data Teams
Redshift in ChatGPT Work: governed conversational analytics
The analytical interface is beginning to move outside the BI application while keeping governed enterprise data behind it.
Cloud Platforms
Redshift and Databricks: querying across platforms
Interoperability is becoming a practical alternative to consolidating every analytical workload onto one platform.
Data Engineering
Snowflake Zero-Copy Interactive: fewer serving copies
Low-latency analytical workloads increasingly do not require teams to create and maintain another specialised copy of their data.
Modern Data Stack
The data stack is becoming agent-ready
September’s platform changes point to a common architectural challenge: giving agents reliable context, explicit permissions and controlled execution.
Modern Data Stack
AWS’s agentic lakehouse: data access and governance
AWS’s multi-cloud lakehouse pattern puts catalogue access and permissions at the centre of agent design.
Cloud Platforms
AWS and Salesforce: governing zero-copy data access
AWS describes direct access from Salesforce Data 360 to Iceberg data in S3. Shared access still needs explicit ownership.
BI & Analytics
BigQuery AI.AGG: summarisation, quality and cost
AI.AGG brings grouped text and image summarisation into BigQuery. Test the summaries as carefully as the query.
Data Governance
Gemini Enterprise logs: what data teams should monitor
Google’s BigQuery logging pattern supports deeper analysis of Gemini Enterprise use. The telemetry also needs its own controls.
Data Engineering
Snowflake Cortex AI Function Studio: quality and cost
Snowflake’s AI Function Studio evaluates quality and inference cost together. A vendor benchmark is a starting point for your own test.
Data Engineering
Snowflake SCD-1 pipelines: partial updates and nulls
Partial change events can erase valid values if a current-state pipeline treats missing fields as replacements.
AI for Data Teams
Maia Foundation on BigQuery: what to evaluate
Matillion announced Maia Foundation for BigQuery. Teams should evaluate generated pipelines against their own delivery requirements.