Apache Spark
3 mentions across all digests
Open-source distributed data processing framework used as the foundation for Lakeflow Declarative Pipelines.
Definity embeds agents inside Spark pipelines to catch failures before they reach agentic AI systems
Definity embeds agents directly into Apache Spark pipelines to catch data quality failures before corrupted inputs reach downstream agentic AI systems.
Agentic Data Engineering with Genie Code and Lakeflow
Databricks launches Genie Code, an agentic assistant that generates production-ready Spark data pipelines from natural language, cutting development time from weeks to hours while maintaining governance through Unity Catalog integration.
New cascade field for deleting Unity Catalog pipelines (Beta)
Databricks adds cascade deletion and RAG-optimized ai_prep_search in April release alongside Spark 4.1.0 runtime update.