org.apache.spark.sql.parquet

Class InsertIntoParquetTable

Object

org.apache.spark.sql.catalyst.trees.TreeNode<PlanType>

org.apache.spark.sql.catalyst.plans.QueryPlan<SparkPlan>

org.apache.spark.sql.execution.SparkPlan

org.apache.spark.sql.parquet.InsertIntoParquetTable

All Implemented Interfaces:

com.typesafe.scalalogging.slf4j.Logging, java.io.Serializable, scala.Equals, scala.Product

public class InsertIntoParquetTable
extends SparkPlan
implements scala.Product, scala.Serializable

Operator that acts as a sink for queries on RDDs and can be used to store the output inside a directory of Parquet files. This operator is similar to Hive's INSERT INTO TABLE operation in the sense that one can choose to either overwrite or append to a directory. Note that consecutive insertions to the same table must have compatible (source) schemas.

WARNING: EXPERIMENTAL! InsertIntoParquetTable with overwrite=false may cause data corruption in the case that multiple users try to append to the same table simultaneously. Inserting into a table that was previously generated by other means (e.g., by creating an HDFS directory and importing Parquet files generated by other tools) may cause unpredicted behaviour and therefore results in a RuntimeException (only detected via filename pattern so will not catch all cases).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InsertIntoParquetTable(org.apache.spark.sql.parquet.ParquetRelation relation, SparkPlan child, boolean overwrite, SQLContext sqlContext)

Method Summary

Methods

Modifier and Type	Method and Description
SparkPlan	child()
RDD<org.apache.spark.sql.catalyst.expressions.Row>	execute() Inserts all rows into the Parquet file.
scala.collection.immutable.List<SQLContext>	otherCopyArgs()
scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Attribute>	output()
boolean	overwrite()
org.apache.spark.sql.parquet.ParquetRelation	relation()
SQLContext	sqlContext()

Methods inherited from class org.apache.spark.sql.execution.SparkPlan

executeCollect, outputPartitioning, requiredChildDistribution

Methods inherited from class org.apache.spark.sql.catalyst.plans.QueryPlan

expressions, generateSchemaString, generateSchemaString, org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionDown$1, org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionUp$1, outputSet, printSchema, schemaString, transformAllExpressions, transformExpressions, transformExpressionsDown, transformExpressionsUp

Methods inherited from class org.apache.spark.sql.catalyst.trees.TreeNode

apply, argString, asCode, children, collect, fastEquals, flatMap, foreach, generateTreeString, getNodeNumbered, id, makeCopy, map, mapChildren, nextId, nodeName, numberedTreeString, sameInstance, simpleString, stringArgs, toString, transform, transformChildrenDown, transformChildrenUp, transformDown, transformUp, treeString, withNewChildren

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface scala.Product

productArity, productElement, productIterator, productPrefix

Methods inherited from interface scala.Equals

canEqual, equals

Methods inherited from interface com.typesafe.scalalogging.slf4j.Logging

logger

Constructor Detail

InsertIntoParquetTable

public InsertIntoParquetTable(org.apache.spark.sql.parquet.ParquetRelation relation,
                      SparkPlan child,
                      boolean overwrite,
                      SQLContext sqlContext)

Method Detail

relation

public org.apache.spark.sql.parquet.ParquetRelation relation()

child

public SparkPlan child()

overwrite

public boolean overwrite()

sqlContext

public SQLContext sqlContext()

execute

public RDD<org.apache.spark.sql.catalyst.expressions.Row> execute()

Inserts all rows into the Parquet file.

Specified by:

execute in class SparkPlan

output

public scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Attribute> output()

Specified by:

output in class org.apache.spark.sql.catalyst.plans.QueryPlan<SparkPlan>

otherCopyArgs

public scala.collection.immutable.List<SQLContext> otherCopyArgs()