Vibe Check: Can AI Actually Write Spark? Live Coding w/ Databricks & Snowflake

Holden Karau (Snowflake, formerly Databricks) and Lisa Cao (Databricks DevRel) joined us to dig into what it actually looks like to write Spark code with AI agents in 2026. 

We worked through a real example from Holden's High Performance Spark Second Edition, a Goldilocks-style exact percentiles problem across hundreds of columns, and watched Claude Code handle it from a cold-start repo, then again with proper context and a reference solution. 

Lisa walked through using a skill file to get Claude to write declarative pipelines the right way instead of hallucinating the syntax, and we talked about why local mode, Arrow UDFs, and the upcoming Python UDF transpilation work in Spark 4.3 matter for AI-assisted Spark development.

We also covered: why the "agents trying to shut you up" failure mode shows up in code generation, the case for property-based testing as a guardrail for agentic coding, Spark 3 to Spark 4 porting as a more reliable path than one-shotting Spark 4, and where agents still struggle (dependency management, Spark internals, real stack trace debugging vs. pattern matching from Stack Overflow).

Resources:
- Apache Spark: https://spark.apache.org & https://github.com/apache/spark
- High Performance Spark (O'Reilly): https://www.oreilly.com
- Spark testing base: https://github.com/holdenk/spark-testing-base
- Databricks Spark: https://www.databricks.com/spark/about
- Data + AI Summit (June 15 to 18, San Francisco): https://www.databricks.com/dataaisummit
- Snowflake Summit (June 1 to 4, San Francisco): https://www.snowflake.com/summit
- PySpark Data Sources: https://github.com/allisonwang-db/pyspark-data-sources
- PySpark SDP: https://github.com/lisancao/pyspark-sdp
- Spark Testing Base: https://github.com/holdenk/spark-testing-base
- Distributed Computing 4 Kids: https://distributedcomputing4kids.com/

What's working for you when you write Spark with agents? Drop your patterns and gotchas below. And let us know what you want to see on future episodes.
---

Temporal is the simple, scalable, open source way to write and run reliable cloud applications. 

Learn more
Blog: https://temporal.io/blog
How Temporal Works: https://temporal.io/how-temporal-works
Community Slack: https://temporal.io/slack

Developer resources
Docs: https://docs.temporal.io
Courses: https://learn.temporal.io/courses
Support forum: https://community.temporal.io