org.apache.spark.sql.sources.v2.reader

Interface InputPartition<T>

All Superinterfaces:

java.io.Serializable

All Known Subinterfaces:

ContinuousInputPartition<T>

@InterfaceStability.Evolving
public interface InputPartition<T>
extends java.io.Serializable

An input partition returned by DataSourceReader.planInputPartitions() and is responsible for creating the actual data reader of one RDD partition. The relationship between InputPartition and InputPartitionReader is similar to the relationship between Iterable and Iterator. Note that InputPartitions will be serialized and sent to executors, then InputPartitionReaders will be created on executors to do the actual reading. So InputPartition must be serializable while InputPartitionReader doesn't need to be.

Method Summary

Modifier and Type	Method and Description
InputPartitionReader<T>	createPartitionReader() Returns an input partition reader to do the actual reading work.
default String[]	preferredLocations() The preferred locations where the input partition reader returned by this partition can run faster, but Spark does not guarantee to run the input partition reader on these locations.

Method Detail

preferredLocations

default String[] preferredLocations()

The preferred locations where the input partition reader returned by this partition can run faster, but Spark does not guarantee to run the input partition reader on these locations. The implementations should make sure that it can be run on any location. The location is a string representing the host name. Note that if a host name cannot be recognized by Spark, it will be ignored as it was not in the returned locations. The default return value is empty string array, which means this input partition's reader has no location preference. If this method fails (by throwing an exception), the action will fail and no Spark job will be submitted.

createPartitionReader

InputPartitionReader<T> createPartitionReader()

Returns an input partition reader to do the actual reading work. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.