org.hackystat.sensorshell
Class MultiSensorShell

java.lang.Object
  extended by org.hackystat.sensorshell.MultiSensorShell
All Implemented Interfaces:
Shell

public class MultiSensorShell
extends java.lang.Object
implements Shell

MultiSensorShell is a wrapper around SingleSensorShell that is designed for high performance transmission of sensor data instances from a client to a server. Prior research has determined that when a single SensorShell is used to transmit a large amount of data in a short period of time, it can spend a substantial portion of its time blocked while waiting for an HTTP PUT to complete. MultiSensorShell overcomes this problem by instantiating multiple SensorShell instances internally and then passing sensor data to them in a round-robin fashion. Each SensorShell is passed an autoSendTimeInterval value, which results in a separate thread for each SensorShell instance that will concurrently send any buffered data at regular time intervals. This significantly reduces blocked time for MultiSensorShell, because when any individual SensorShell instance is performing its HTTP PUT call, the MultiSensorShell can be concurrently adding data to one of the other SensorShell instances.

The sensorshell.properties file provides a number of tuning parameters for MultiSensorShell processing. We currently recommend the following settings for best performance:

Note that offline storage and recovery are automatically disabled when multishell is enabled.

The TestMultiSensorShell class provides a main() method that we have used to do some simple performance evaluation, which we report on next. All results were obtained using a MacBook Pro with a 2.33 Ghz Intel Core Duo processor and 3 GB of 667 Mhz DDR2 SDRAM. Both the client and SensorBase server were running on this computer to minimize network latency issues.

If you instantiate a MultiSensorShell with the number of SensorShells set to 1, you effectively get the default case. In this situation, we have found that the average time to send a single SensorData instance is approximately 6 milliseconds, almost independent of the settings for batchSize and the autoSendInterval. Increasing the number of SensorShell instances to 5 doubles the throughput, to approximately 3 milliseconds per instance. At this point, some kind of performance plateau is reached, with further tweaking of the tuning parameters seeming to have little effect. We do not know whether this is a "real" limit or an artificial limit based upon some environmental feature.

With the sensorshell.properties settings listed above, we have achieved throughput of 2.8 milliseconds per instance (which is equivalent to 360 instances per second and 1.2M instances per hour.)

We have also found that we can store around 350,000 sensor data instances per GB of disk space.

Note that we are effectively disabling autosend.batchsize by setting it to a high value (30,000). This is because reaching the batchSize limit forces a blocking send() of the data, which is precisely what we want to avoid in MultiSensorShell. Instead, we try to tune the autosend.timeinterval so that as many of our send() invocations as possible occur asynchronously in a separate thread.

Note that a single SensorShell instance is simpler, creates less processing overhead, and has equivalent performance to MultiSensorShell for transmission loads up to a dozen or so sensor data instances per second. We recommend using a single SensorShell instance rather than MultiSensorShell unless optimizing data transmission throughput is an important requirement.

Author:
Philip Johnson

Constructor Summary
MultiSensorShell(SensorShellProperties properties, java.lang.String toolName)
          Creates a new MultiSensorShell for multi-threaded transmission of SensorData instances to a SensorBase.
 
Method Summary
 void add(java.util.Map<java.lang.String,java.lang.String> keyValMap)
          Converts the values in the KeyValMap to a SensorData instance and adds it to the Shell.
 void add(org.hackystat.sensorbase.resource.sensordata.jaxb.SensorData sensorData)
          Adds the passed SensorData instance to the Shell.
 SensorShellProperties getProperties()
          Returns the SensorShell properties instance used to create this SensorShell.
 boolean hasOfflineData()
          Returns true if any of shells in this interface have stored data offline.
 boolean ping()
          Returns true if the host can be pinged and the email/password combination is valid.
 void quit()
          Invokes quit() on this Shell, which invokes a final send() and closes any logging files.
 int send()
          Immediately invokes send() on this Shell.
 void statechange(long resourceCheckSum, java.util.Map<java.lang.String,java.lang.String> keyValMap)
          Implements the "StateChange" algorithm.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

MultiSensorShell

public MultiSensorShell(SensorShellProperties properties,
                        java.lang.String toolName)
Creates a new MultiSensorShell for multi-threaded transmission of SensorData instances to a SensorBase.

Parameters:
properties - A SensorProperties instance.
toolName - The name of the tool, used to name the log file.
Method Detail

add

public void add(org.hackystat.sensorbase.resource.sensordata.jaxb.SensorData sensorData)
         throws SensorShellException
Adds the passed SensorData instance to the Shell.

Specified by:
add in interface Shell
Parameters:
sensorData - The SensorData instance to be queued for transmission.
Throws:
SensorShellException - If problems occur sending the data.

add

public void add(java.util.Map<java.lang.String,java.lang.String> keyValMap)
         throws SensorShellException
Converts the values in the KeyValMap to a SensorData instance and adds it to the Shell. Owner will default to the hackystat user in the sensor.properties file. Timestamp and Runtime will default to the current time.

Specified by:
add in interface Shell
Parameters:
keyValMap - The map of key-value pairs.
Throws:
SensorShellException - If the Map cannot be translated into SensorData, typically because a value was passed for Timestamp or Runtime that could not be parsed into XMLGregorianCalendar. Or if problems occur sending the data.

ping

public boolean ping()
Returns true if the host can be pinged and the email/password combination is valid.

Specified by:
ping in interface Shell
Returns:
True if the host can be pinged and the user credentials are valid.

send

public int send()
         throws SensorShellException
Immediately invokes send() on this Shell. Note that you will rarely want to invoke this method. Instead, during normal operation you will rely on the autoSendTimeInterval to invoke send() in a separate thread, and then invoke quit() to invoke send() at the conclusion of the run.

Specified by:
send in interface Shell
Returns:
The total number of instances sent by Shell.
Throws:
SensorShellException - If problems occur sending the data.

quit

public void quit()
          throws SensorShellException
Invokes quit() on this Shell, which invokes a final send() and closes any logging files.

Specified by:
quit in interface Shell
Throws:
SensorShellException - If an exception occurred during any autosend.

hasOfflineData

public boolean hasOfflineData()
Returns true if any of shells in this interface have stored data offline.

Specified by:
hasOfflineData in interface Shell
Returns:
True if any of these shells have stored offline data during this session.

statechange

public void statechange(long resourceCheckSum,
                        java.util.Map<java.lang.String,java.lang.String> keyValMap)
                 throws java.lang.Exception
Implements the "StateChange" algorithm. The goal of StateChange is to add a new SensorData instance for sending to the SensorBase only when "something has changed" in the tool (which is typically an interactive editor or IDE). This method is designed to be used in conjunction with a timer-based process in the sensor client. The timer-based process should wake up at regular intervals, determine the currently "active" resource (typically a file), and compute a "checksum" for that resource representing its state. (This checksum is typically the size of the resource in bytes or characters.) Having obtained those two values, the sensor client can then create a keyValMap as if it were going to do a regular Add command, and invoke this method with it as well as with the resourceCheckSum. This method keeps track of the last Resource and last ResourceCheckSum provided to it, and if either has changed, it will automatically invoke the Add command, passing the keyValMap to it.

Thus, if an editor is running but the user is out at lunch, repeated invocations of the StateChange method will not result in any new data being sent to the server.

Specified by:
statechange in interface Shell
Parameters:
resourceCheckSum - An integer representing the state of the Resource.
keyValMap - A map of key-value pairs representing sensor data fields and properties.
Throws:
java.lang.Exception - If problems occur during the Add (if the Add actually occurs.)

getProperties

public SensorShellProperties getProperties()
Returns the SensorShell properties instance used to create this SensorShell.

Specified by:
getProperties in interface Shell
Returns:
The SensorShellProperties instance.