org.apache.spark.sql

Class CacheManager

Object

org.apache.spark.sql.CacheManager

All Implemented Interfaces:

Logging

public class CacheManager
extends Object
implements Logging

Provides support in a SQLContext for caching query results and automatically using these cached results when subsequent queries are executed. Data is cached using byte buffers stored in an InMemoryRelation. This relation is automatically substituted query plans that return the sameResult as the originally cached query.

Internal to Spark SQL.

Constructor Summary

Constructors

Constructor and Description
CacheManager(SQLContext sqlContext)

Method Summary

Methods

Modifier and Type	Method and Description
void	cacheQuery(DataFrame query, scala.Option<String> tableName, StorageLevel storageLevel) Caches the data produced by the logical representation of the given schema rdd.
void	cacheTable(String tableName) Caches the specified table in-memory.
void	clearCache() Clears all cached tables.
void	invalidateCache(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Invalidates the cache of any data that contains plan.
boolean	isCached(String tableName) Returns true if the table is currently cached in-memory.
boolean	isEmpty() Checks if the cache is empty.
scala.Option<CachedData>	lookupCachedData(DataFrame query) Optionally returns cached data for the given DataFrame
scala.Option<CachedData>	lookupCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Optionally returns cached data for the given LogicalPlan.
boolean	tryUncacheQuery(DataFrame query, boolean blocking) Tries to remove the data for the given DataFrame from the cache if it's cached
void	uncacheQuery(DataFrame query, boolean blocking) Removes the data for the given DataFrame from the cache
void	uncacheTable(String tableName) Removes the specified table from the in-memory cache.
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	useCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Replaces segments of the given logical plan with cached versions where possible.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

CacheManager

public CacheManager(SQLContext sqlContext)

Method Detail

isCached

public boolean isCached(String tableName)

Returns true if the table is currently cached in-memory.

cacheTable

public void cacheTable(String tableName)

Caches the specified table in-memory.

uncacheTable

public void uncacheTable(String tableName)

Removes the specified table from the in-memory cache.

clearCache

public void clearCache()

Clears all cached tables.

isEmpty

public boolean isEmpty()

Checks if the cache is empty.

cacheQuery

public void cacheQuery(DataFrame query,
              scala.Option<String> tableName,
              StorageLevel storageLevel)

Caches the data produced by the logical representation of the given schema rdd. Unlike RDD.cache(), the default storage level is set to be MEMORY_AND_DISK because recomputing the in-memory columnar representation of the underlying table is expensive.

uncacheQuery

public void uncacheQuery(DataFrame query,
                boolean blocking)

Removes the data for the given DataFrame from the cache

tryUncacheQuery

public boolean tryUncacheQuery(DataFrame query,
                      boolean blocking)

Tries to remove the data for the given DataFrame from the cache if it's cached

lookupCachedData

public scala.Option<CachedData> lookupCachedData(DataFrame query)

Optionally returns cached data for the given DataFrame

lookupCachedData

public scala.Option<CachedData> lookupCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Optionally returns cached data for the given LogicalPlan.

useCachedData

public org.apache.spark.sql.catalyst.plans.logical.LogicalPlan useCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Replaces segments of the given logical plan with cached versions where possible.

invalidateCache

public void invalidateCache(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Invalidates the cache of any data that contains plan. Note that it is possible that this function will over invalidate.