org.apache.spark.sql.sources.v2.reader

Interface InputPartitionReader<T>

All Superinterfaces:

AutoCloseable, java.io.Closeable

All Known Subinterfaces:

ContinuousInputPartitionReader<T>

@InterfaceStability.Evolving
public interface InputPartitionReader<T>
extends java.io.Closeable

An input partition reader returned by InputPartition.createPartitionReader() and is responsible for outputting data for a RDD partition. Note that, Currently the type `T` can only be InternalRow for normal data source readers, ColumnarBatch for data source readers that mix in SupportsScanColumnarBatch.

Method Summary

Modifier and Type	Method and Description
T	get() Return the current record.
boolean	next() Proceed to next record, returns false if there is no more records.

Methods inherited from interface java.io.Closeable

close

Method Detail

next

boolean next()
      throws java.io.IOException

Proceed to next record, returns false if there is no more records. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.

Throws:

java.io.IOException - if failure happens during disk/network IO like reading files.

get

T get()

Return the current record. This method should return same value until `next` is called. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.