Robin Mode Failure Analysis

This document summarizes failure reasons when running the Sparkless test suite with the Robin backend (SPARKLESS_TEST_BACKEND=robin, SPARKLESS_BACKEND=robin), buckets them by cause, and classifies ownership (Sparkless vs robin-sparkless).

Reference run: Full suite (see robin_mode_test_report.md): 1,640 FAILED, 867 PASSED, 21 SKIPPED, 7 ERROR.
Sample run: A subset of failing tests plus all 7 parquet ERROR tests was run with --tb=long; output is in tests/robin_failures_sample.txt.

1. Counts by bucket and exception type

Full-suite totals (from reference run)

Result

Count

FAILED

1,640

ERROR

7

PASSED

867

SKIPPED

21

Sample-based exception breakdown (from robin_failures_sample.txt)

From the sampled FAILED tests and the 7 ERROR tests:

Bucket

Exception / cause

Sample count

Owner

A

SparkUnsupportedOperationError / “Backend ‘robin’ does not support these operations”

8 of 12 FAILED in sample

Sparkless (policy)

A

QueryExecutionException (wraps same reason)

1

Sparkless

B

ValueError: RobinMaterializer PoC only supports schema with exactly 3 columns

1

Sparkless (or robin-sparkless if API limits schema)

C

AssertionError (wrong row count / wrong result)

2 (sample) + 7 (parquet tests when run without ERROR)

Sparkless or robin-sparkless (see below)

C

TypeError: '>' not supported between instances of 'builtins.Column' and 'builtins.Column'

2

Sparkless (comparison/row handling)

C

AssertionError: assert 'robin' in ['mock', 'pyspark']

1

Sparkless (test/fixture expectation)

D

AttributeError: 'RobinStorageManager' object has no attribute 'db_path' (teardown)

7 (all ERRORs)

Sparkless (Robin storage delegate)

Note: The 7 parquet tests both FAIL (AssertionError on row counts) and ERROR (teardown db_path). So the 7 ERRORs are teardown failures; the same tests also fail in the test body with wrong results.

2. Bucket descriptions and ownership

Bucket A – SparkUnsupportedOperationError / “does not support these operations”

  • Cause: Robin materializer’s can_handle_operations() returns False for the op set; Sparkless raises (fail-fast).

  • Owner: Sparkless (fail-fast policy).

  • Sample ops seen: select, union, filter, join, orderBy, and combinations.

Options for Sparkless:
(1) Keep fail-fast and document;
(2) Extend the Robin materializer to support more ops (limited by what robin_sparkless exposes).

Bucket B – ValueError / “exactly 3 columns”

  • Cause: Robin materializer’s strict 3-column PoC schema (RobinMaterializer only supports that shape for robin_sparkless.create_dataframe).

  • Owner: Sparkless if we can relax the schema in our materializer when robin_sparkless supports it; robin-sparkless if their create_dataframe only supports that shape (feature request there).

Bucket C – AssertionError / wrong result / TypeError (Column comparison)

  • Wrong row count / wrong result: If the failure happens after calling robin_sparkless (e.g. after materialize() returns), the wrong result is likely robin-sparkless (engine or API). If it happens in Sparkless (schema handling, row conversion, storage append semantics), Sparkless.

  • Parquet append tests: Same 7 tests fail with AssertionError (e.g. “Table should have 1 row after append, got 2”). This points to Sparkless storage/session semantics (e.g. how Robin storage delegate handles append/visibility) or test expectations.

  • TypeError Column vs Column: Comparison or ordering of Sparkless Column objects that are not simplified before reaching Robin or internal comparison → Sparkless (comparison/row handling or test helper).

  • assert 'robin' in ['mock', 'pyspark']: Test or fixture assumes only mock/pyspark backends → Sparkless (test/fixture update).

Bucket D – ERROR (setup/teardown)

  • Cause: All 7 ERRORs are AttributeError: ‘RobinStorageManager’ object has no attribute ‘db_path’ in test teardown. The test file accesses self.spark._storage.db_path; RobinStorageManager delegates to Polars storage but does not expose a db_path attribute.

  • Owner: Sparkless (Robin storage delegate compatibility).

  • Fix: Add a db_path property (or attribute) to RobinStorageManager that mirrors PolarsStorageManager.db_path (e.g. expose the same path used when constructing the delegate).

3. Sparkless-fixable items

  1. Robin storage delegate: Add db_path to RobinStorageManager (e.g. self._db_path = db_path or ":memory:" and a db_path property) so tests/fixtures that use spark._storage.db_path do not raise in teardown. This clears the 7 ERRORs.

  2. Use robin_sparkless arbitrary schema: Robin-sparkless already provides create_dataframe_from_rows(data, schema) for arbitrary column names and types. Extend the Robin materializer to use it (instead of only create_dataframe for 3 columns) so we support any schema.

  3. Extend Robin materializer to more operations: Robin-sparkless already exposes with_column, order_by, group_by, join, union, and GroupedData aggregations. Extend the Sparkless materializer to translate and call these (in addition to filter/select/limit) so more workloads run on the Robin backend.

  4. Parquet/table append semantics: If append/visibility behavior under Robin storage is wrong, fix storage/session logic for the Robin delegate.

  5. Test/fixture expectations: Update tests that assume only ['mock', 'pyspark'] (e.g. include 'robin' where appropriate).

  6. Column comparison / TypeError: Avoid comparing or ordering unresolved Column objects in code paths used by Robin or in assertion helpers; resolve to values or use supported expressions.

4. robin-sparkless items (for upstream)

Finding (from examining robin-sparkless): The robin-sparkless package already provides the APIs we need:

  • Arbitrary schema: create_dataframe_from_rows(data, schema) accepts data (list of dicts or lists) and schema as a list of (name, dtype_str) (e.g. [("id", "bigint"), ("name", "string")]). Supported dtypes include bigint, double, string, boolean, date, timestamp, etc. Only create_dataframe() is 3-column (i64, i64, str); the flexible entry point is create_dataframe_from_rows.

  • Operations: The DataFrame API already exposes filter, select, with_column, order_by, order_by_exprs, group_by, limit, union, union_by_name, join, and GroupedData (count, sum, avg, min, max, agg, etc.). See robin-sparkless robin_sparkless.pyi and Rust src/dataframe/ (transformations, joins, aggregations).

So no upstream feature requests are needed for “more operations” or “flexible schema.” The gap is on the Sparkless side: our Robin materializer currently uses only create_dataframe (3-column) and supports only filter/select/limit. We should:

  1. Use create_dataframe_from_rows when the schema is not exactly 3 columns (or always, for consistency), converting Sparkless StructType and row data into the format robin_sparkless expects.

  2. Extend the materializer to translate and invoke more operations (withColumn, groupBy, orderBy, join, union, etc.) using the existing robin_sparkless DataFrame API.

Bugs: Any concrete bug in robin_sparkless (wrong result, missing API, signature mismatch) should still be reported upstream with repro and environment. See robin_sparkless_issues.md for issue templates.

5. How to re-run this analysis

  1. Full Robin run (summary):

    pip install robin-sparkless  # if needed
SPARKLESS_TEST_BACKEND=robin SPARKLESS_BACKEND=robin python -m pytest tests/ --ignore=tests/archive -n 10 -v --tb=short 2>&1 | tee tests/robin_mode_test_results.txt

  2. Sample of failing tests with long tracebacks:

    SPARKLESS_TEST_BACKEND=robin SPARKLESS_BACKEND=robin python scripts/run_robin_failure_sample.py --max 50 --output tests/robin_failures_sample.txt

  3. Only the 7 parquet ERROR tests:

    SPARKLESS_TEST_BACKEND=robin SPARKLESS_BACKEND=robin python scripts/run_robin_failure_sample.py --parquet-only --output tests/robin_failures_sample.txt

  4. Parse exception types: Inspect tests/robin_failures_sample.txt for lines starting with FAILED or ERROR; the text after the last - is the exception type and message. Count by exception type and message to refresh the table in §1.

6. Files touched / added

  • Report: docs/robin_mode_failure_analysis.md (this file).

  • Sample output: tests/robin_failures_sample.txt (appended by the script).

  • Script: scripts/run_robin_failure_sample.py (runs a fixed list of failing test IDs with --tb=long and appends to a file).