org.apache.spark.streaming.dstream

Class InputDStream<T>

java.lang.Object

org.apache.spark.streaming.dstream.DStream<T>

org.apache.spark.streaming.dstream.InputDStream<T>

All Implemented Interfaces:

java.io.Serializable, Logging

Direct Known Subclasses:

ConstantInputDStream, ReceiverInputDStream

public abstract class InputDStream<T>
extends DStream<T>

This is the abstract base class for all input streams. This class provides methods start() and stop() which is called by Spark Streaming system to start and stop receiving data. Input streams that can generate RDDs from new data by running a service/thread only on the driver node (that is, without running a receiver on worker nodes), can be implemented by directly inheriting this InputDStream. For example, FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for new files and generates RDDs with the new files. For implementing input streams that requires running a receiver on the worker nodes, use ReceiverInputDStream as the parent class.

param: ssc_ Streaming context that will execute this input stream

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InputDStream(StreamingContext ssc_, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
protected scala.Option<java.lang.String>	baseScope() The base scope associated with the operation that created this DStream.
scala.collection.immutable.List<DStream<?>>	dependencies() List of parent DStreams on which this DStream depends on
int	id() This is an unique identifier for the input stream.
Time	lastValidTime()
protected scala.Option<org.apache.spark.streaming.scheduler.RateController>	rateController()
Duration	slideDuration() Time interval after which the DStream generates a RDD
abstract void	start() Method called to start receiving data.
abstract void	stop() Method called to stop receiving data.

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, checkpointData, checkpointDuration, compute, context, count, countByValue, countByValueAndWindow, countByWindow, createRDDWithLocalProperties, creationSite, filter, flatMap, foreach, foreach, foreachRDD, foreachRDD, generatedRDDs, glom, graph, map, mapPartitions, mustCheckpoint, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, rememberDuration, repartition, saveAsObjectFiles, saveAsTextFiles, slice, slice, ssc, storageLevel, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, window, window, zeroTime

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

InputDStream

public InputDStream(StreamingContext ssc_,
            scala.reflect.ClassTag<T> evidence$1)

Method Detail

lastValidTime

public Time lastValidTime()

id

public int id()

This is an unique identifier for the input stream.

rateController

protected scala.Option<org.apache.spark.streaming.scheduler.RateController> rateController()

baseScope

protected scala.Option<java.lang.String> baseScope()

The base scope associated with the operation that created this DStream.

For InputDStreams, we use the name of this DStream as the scope name. If an outer scope is given, we assume that it includes an alternative name for this stream.

Overrides:

baseScope in class DStream<T>

Returns:

(undocumented)

dependencies

public scala.collection.immutable.List<DStream<?>> dependencies()

Description copied from class: DStream

List of parent DStreams on which this DStream depends on

Specified by:

dependencies in class DStream<T>

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates a RDD

Specified by:

slideDuration in class DStream<T>

start

public abstract void start()

Method called to start receiving data. Subclasses must implement this method.

stop

public abstract void stop()

Method called to stop receiving data. Subclasses must implement this method.