org.apache.spark.sql.internal

Class CatalogImpl

Object

org.apache.spark.sql.catalog.Catalog

org.apache.spark.sql.internal.CatalogImpl

public class CatalogImpl
extends Catalog

Internal implementation of the user-facing Catalog.

Constructor Summary

Constructors

Constructor and Description
CatalogImpl(SparkSession sparkSession)

Method Summary

Methods

Modifier and Type	Method and Description
void	cacheTable(String tableName) Caches the specified table in-memory.
void	clearCache() Removes all cached tables from the in-memory cache.
Dataset<Row>	createExternalTable(String tableName, String path) :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String source, java.util.Map<String,String> options) :: Experimental :: Creates an external table from the given path based on a data source and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.
Dataset<Row>	createExternalTable(String tableName, String path, String source) :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, java.util.Map<String,String> options) :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.
String	currentDatabase() Returns the current default database in this session.
boolean	databaseExists(String dbName) Check if the database with the specified name exists.
boolean	dropGlobalTempView(String viewName) Drops the global temporary view with the given view name in the catalog.
boolean	dropTempView(String viewName) Drops the local temporary view with the given view name in the catalog.
boolean	functionExists(String functionName) Check if the function with the specified name exists.
boolean	functionExists(String dbName, String functionName) Check if the function with the specified name exists in the specified database.
Database	getDatabase(String dbName) Get the database with the specified name.
Function	getFunction(String functionName) Get the function with the specified name.
Function	getFunction(String dbName, String functionName) Get the function with the specified name.
Table	getTable(String tableName) Get the table or view with the specified name.
Table	getTable(String dbName, String tableName) Get the table or view with the specified name in the specified database.
boolean	isCached(String tableName) Returns true if the table is currently cached in-memory.
Dataset<Column>	listColumns(String tableName) Returns a list of columns for the given table in the current database.
Dataset<Column>	listColumns(String dbName, String tableName) Returns a list of columns for the given table in the specified database.
Dataset<Database>	listDatabases() Returns a list of databases available across all sessions.
Dataset<Function>	listFunctions() Returns a list of functions registered in the current database.
Dataset<Function>	listFunctions(String dbName) Returns a list of functions registered in the specified database.
Dataset<Table>	listTables() Returns a list of tables in the current database.
Dataset<Table>	listTables(String dbName) Returns a list of tables in the specified database.
static <T extends org.apache.spark.sql.catalyst.DefinedByConstructorParams> Dataset<T>	makeDataset(scala.collection.Seq<T> data, SparkSession sparkSession, scala.reflect.api.TypeTags.TypeTag<T> evidence$1)
void	refreshByPath(String resourcePath) Refresh the cache entry and the associated metadata for all dataframes (if any), that contain the given data source path.
void	refreshTable(String tableName) Refresh the cache entry for a table, if any.
void	setCurrentDatabase(String dbName) Sets the current default database in this session.
boolean	tableExists(String tableName) Check if the table or view with the specified name exists.
boolean	tableExists(String dbName, String tableName) Check if the table or view with the specified name exists in the specified database.
void	uncacheTable(String tableName) Removes the specified table from the in-memory cache.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CatalogImpl

public CatalogImpl(SparkSession sparkSession)

Method Detail

makeDataset

public static <T extends org.apache.spark.sql.catalyst.DefinedByConstructorParams> Dataset<T> makeDataset(scala.collection.Seq<T> data,
                                                                                          SparkSession sparkSession,
                                                                                          scala.reflect.api.TypeTags.TypeTag<T> evidence$1)

currentDatabase

public String currentDatabase()

Returns the current default database in this session.

Specified by:

currentDatabase in class Catalog

Returns:

(undocumented)

setCurrentDatabase

public void setCurrentDatabase(String dbName)
                        throws AnalysisException

Sets the current default database in this session.

Specified by:

setCurrentDatabase in class Catalog

Parameters:

dbName - (undocumented)

Throws:

AnalysisException

listDatabases

public Dataset<Database> listDatabases()

Returns a list of databases available across all sessions.

Specified by:

listDatabases in class Catalog

Returns:

(undocumented)

listTables

public Dataset<Table> listTables()

Returns a list of tables in the current database. This includes all temporary tables.

Specified by:

listTables in class Catalog

Returns:

(undocumented)

listTables

public Dataset<Table> listTables(String dbName)
                          throws AnalysisException

Returns a list of tables in the specified database. This includes all temporary tables.

Specified by:

listTables in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listFunctions

public Dataset<Function> listFunctions()

Returns a list of functions registered in the current database. This includes all temporary functions

Specified by:

listFunctions in class Catalog

Returns:

(undocumented)

listFunctions

public Dataset<Function> listFunctions(String dbName)
                                throws AnalysisException

Returns a list of functions registered in the specified database. This includes all temporary functions

Specified by:

listFunctions in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listColumns

public Dataset<Column> listColumns(String tableName)
                            throws AnalysisException

Returns a list of columns for the given table in the current database.

Specified by:

listColumns in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listColumns

public Dataset<Column> listColumns(String dbName,
                          String tableName)
                            throws AnalysisException

Returns a list of columns for the given table in the specified database.

Specified by:

listColumns in class Catalog

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

getDatabase

public Database getDatabase(String dbName)

Get the database with the specified name. This throws an AnalysisException when no Database can be found.

Specified by:

getDatabase in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

getTable

public Table getTable(String tableName)

Get the table or view with the specified name. This table can be a temporary view or a table/view in the current database. This throws an AnalysisException when no Table can be found.

Specified by:

getTable in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

getTable

public Table getTable(String dbName,
             String tableName)

Get the table or view with the specified name in the specified database. This throws an AnalysisException when no Table can be found.

Specified by:

getTable in class Catalog

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

getFunction

public Function getFunction(String functionName)

Get the function with the specified name. This function can be a temporary function or a function in the current database. This throws an AnalysisException when no Function can be found.

Specified by:

getFunction in class Catalog

Parameters:

functionName - (undocumented)

Returns:

(undocumented)

getFunction

public Function getFunction(String dbName,
                   String functionName)

Get the function with the specified name. This returns None when no Function can be found.

Specified by:

getFunction in class Catalog

Parameters:

dbName - (undocumented)

functionName - (undocumented)

Returns:

(undocumented)

databaseExists

public boolean databaseExists(String dbName)

Check if the database with the specified name exists.

Specified by:

databaseExists in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

tableExists

public boolean tableExists(String tableName)

Check if the table or view with the specified name exists. This can either be a temporary view or a table/view in the current database.

Specified by:

tableExists in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

tableExists

public boolean tableExists(String dbName,
                  String tableName)

Check if the table or view with the specified name exists in the specified database.

Specified by:

tableExists in class Catalog

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

functionExists

public boolean functionExists(String functionName)

Check if the function with the specified name exists. This can either be a temporary function or a function in the current database.

Specified by:

functionExists in class Catalog

Parameters:

functionName - (undocumented)

Returns:

(undocumented)

functionExists

public boolean functionExists(String dbName,
                     String functionName)

Check if the function with the specified name exists in the specified database.

Specified by:

functionExists in class Catalog

Parameters:

dbName - (undocumented)

functionName - (undocumented)

Returns:

(undocumented)

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String path)

:: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

path - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String path,
                               String source)

:: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               java.util.Map<String,String> options)

:: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               java.util.Map<String,String> options)

:: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

dropTempView

public boolean dropTempView(String viewName)

Drops the local temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.

Specified by:

dropTempView in class Catalog

Parameters:

viewName - the name of the view to be dropped.

Returns:

(undocumented)

Since:

2.0.0

dropGlobalTempView

public boolean dropGlobalTempView(String viewName)

Drops the global temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.

Specified by:

dropGlobalTempView in class Catalog

Parameters:

viewName - the name of the view to be dropped.

Returns:

(undocumented)

Since:

2.1.0

isCached

public boolean isCached(String tableName)

Returns true if the table is currently cached in-memory.

Specified by:

isCached in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

cacheTable

public void cacheTable(String tableName)

Caches the specified table in-memory.

Specified by:

cacheTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

uncacheTable

public void uncacheTable(String tableName)

Removes the specified table from the in-memory cache.

Specified by:

uncacheTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

clearCache

public void clearCache()

Removes all cached tables from the in-memory cache.

Specified by:

clearCache in class Catalog

Since:

2.0.0

refreshTable

public void refreshTable(String tableName)

Refresh the cache entry for a table, if any. For Hive metastore table, the metadata is refreshed. For data source tables, the schema will not be inferred and refreshed.

Specified by:

refreshTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

refreshByPath

public void refreshByPath(String resourcePath)

Refresh the cache entry and the associated metadata for all dataframes (if any), that contain the given data source path.

Specified by:

refreshByPath in class Catalog

Parameters:

resourcePath - (undocumented)

Since:

2.0.0