Skip to content

Conversation

@fedimser
Copy link
Contributor

@fedimser fedimser commented Nov 1, 2025

What changes were proposed in this pull request?

  • Created new helper class TwsTester that allows to test user's implementation of Stateful processor in a unit test without running query.
  • Implemented InMemoryStatefulProcessorHandleImpl - in-memory implementation of StatefulProcessorHandle. This implementation is based on InMemoryStatefulProcessorHandleImpl in file BenchmarkTransformWithStateInPySparkStateServer.scala.
  • Added tests for TwsTester.

Why are the changes needed?

To make it possible for users to write unit tests for StatefulProcessor implementation.

Does this PR introduce any user-facing change?

Yes, it adds new public API.

How was this patch tested?

Unit tests in TwsTesterSuite. Ran from sbt with sql/testOnly *TwsTesterSuite

Was this patch authored or co-authored using generative AI tooling?

Yes, used Cursor and claude-4.5-sonnet for writing tests.

@fedimser fedimser marked this pull request as ready for review November 4, 2025 01:29
@fedimser fedimser changed the title [SPARK-54122] Implement TwsTester [SPARK-54122] [WIP] Implement TwsTester Nov 5, 2025
@fedimser fedimser marked this pull request as draft November 5, 2025 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant