org.apache.spark.sql.columnar.compression

Interface CompressibleColumnBuilder<T extends org.apache.spark.sql.types.NativeType>

All Superinterfaces:

ColumnBuilder, Logging

All Known Implementing Classes:

BooleanColumnBuilder, ByteColumnBuilder, DateColumnBuilder, DoubleColumnBuilder, FloatColumnBuilder, IntColumnBuilder, LongColumnBuilder, NativeColumnBuilder, ShortColumnBuilder, StringColumnBuilder, TimestampColumnBuilder

public interface CompressibleColumnBuilder<T extends org.apache.spark.sql.types.NativeType>
extends ColumnBuilder, Logging

A stackable trait that builds optionally compressed byte buffer for a column. Memory layout of the final byte buffer is:

    .--------------------------- Column type ID (4 bytes)
    |   .----------------------- Null count N (4 bytes)
    |   |   .------------------- Null positions (4 x N bytes, empty if null count is zero)
    |   |   |     .------------- Compression scheme ID (4 bytes)
    |   |   |     |   .--------- Compressed non-null elements
    V   V   V     V   V
   +---+---+-----+---+---------+
   |   |   | ... |   | ... ... |
   +---+---+-----+---+---------+
    \-----------/ \-----------/
       header         body
 

Method Summary

Methods

Modifier and Type	Method and Description
void	appendFrom(org.apache.spark.sql.Row row, int ordinal) Appends row(ordinal) to the column builder.
java.nio.ByteBuffer	build() Returns the final columnar byte buffer.
scala.collection.Seq<Encoder<T>>	compressionEncoders()
void	gatherCompressibilityStats(org.apache.spark.sql.Row row, int ordinal)
void	initialize(int initialSize, String columnName, boolean useCompression) Initializes with an approximate lower bound on the expected number of elements in this column.
boolean	isWorthCompressing(Encoder<T> encoder)

Methods inherited from interface org.apache.spark.sql.columnar.ColumnBuilder

columnStats

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Method Detail

compressionEncoders

scala.collection.Seq<Encoder<T>> compressionEncoders()

initialize

void initialize(int initialSize,
              String columnName,
              boolean useCompression)

Description copied from interface: ColumnBuilder

Initializes with an approximate lower bound on the expected number of elements in this column.

Specified by:

initialize in interface ColumnBuilder

isWorthCompressing

boolean isWorthCompressing(Encoder<T> encoder)

gatherCompressibilityStats

void gatherCompressibilityStats(org.apache.spark.sql.Row row,
                              int ordinal)

appendFrom

void appendFrom(org.apache.spark.sql.Row row,
              int ordinal)

Description copied from interface: ColumnBuilder

Appends row(ordinal) to the column builder.

Specified by:

appendFrom in interface ColumnBuilder

build

java.nio.ByteBuffer build()

Description copied from interface: ColumnBuilder

Returns the final columnar byte buffer.

Specified by:

build in interface ColumnBuilder