org.apache.spark.util

Class AccumulatorContext

Object

org.apache.spark.util.AccumulatorContext

public class AccumulatorContext
extends Object

An internal class used to track accumulators by Spark itself.

Constructor Summary

Constructors

Constructor and Description
AccumulatorContext()

Method Summary

Methods

Modifier and Type	Method and Description
static void	clear() Clears all registered AccumulatorV2s.
static scala.Option<AccumulatorV2<?,?>>	get(long id) Returns the AccumulatorV2 registered with the given ID, if any.
static long	newId() Returns a globally unique ID for a new AccumulatorV2.
static int	numAccums() Returns the number of accumulators registered.
static void	register(AccumulatorV2<?,?> a) Registers an AccumulatorV2 created on the driver such that it can be used on the executors.
static void	remove(long id) Unregisters the AccumulatorV2 with the given ID, if any.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AccumulatorContext

public AccumulatorContext()

Method Detail

newId

public static long newId()

Returns a globally unique ID for a new AccumulatorV2. Note: Once you copy the AccumulatorV2 the ID is no longer unique.

Returns:

(undocumented)

numAccums

public static int numAccums()

Returns the number of accumulators registered. Used in testing.

register

public static void register(AccumulatorV2<?,?> a)

Registers an AccumulatorV2 created on the driver such that it can be used on the executors.

All accumulators registered here can later be used as a container for accumulating partial values across multiple tasks. This is what org.apache.spark.scheduler.DAGScheduler does. Note: if an accumulator is registered here, it should also be registered with the active context cleaner for cleanup so as to avoid memory leaks.

If an AccumulatorV2 with the same ID was already registered, this does nothing instead of overwriting it. We will never register same accumulator twice, this is just a sanity check.

Parameters:

a - (undocumented)

remove

public static void remove(long id)

Unregisters the AccumulatorV2 with the given ID, if any.

Parameters:

id - (undocumented)

get

public static scala.Option<AccumulatorV2<?,?>> get(long id)

Returns the AccumulatorV2 registered with the given ID, if any.

Parameters:

id - (undocumented)

Returns:

(undocumented)

clear

public static void clear()

Clears all registered AccumulatorV2s. For testing only.