# Working With Data

## Data Objects Overview

We will call Data Objects in &kst; all objects which appear in the Data Manager. There are ten main kinds of data object in &kst;. The following diagram illustrates the relationships between the different types:

&kst; Data Types

As can be seen in the diagram, the only truly plottable data objects are curves and images. Many data objects contain slave vectors which can be plotted using curves, however — so these data objects are effectively plottable as well.

The usefulness of the Data Objects concept in &kst; is that they can be tied together to create pipelines. An Event Monitor object, for instance, can take the output of another object as its input. Updates to objects in the pipeline propagate automatically. This is one of the key features which makes &kst; powerful for realtime plotting.

### The Data Manager

The Data Manager provides a central location for adding, deleting, and modifying all of the data objects used in &kst;. It can be accessed from Data Data Manager.

The Data Manager
The Data Manager

The panel on the left side of the Data Manager can be used to quickly create new data objects. The Purge button automatically deletes Data Objects which are not Used Data Objects. The Edit and Delete buttons allow you to selectively edit or remove particular objects. The Close button exits the Data Manager.

You can also create new data objects which are based on vectors by right clicking on the corresponding vector and choosing one of the Make ... options. If you select a curve, on the other hand, then you have the option to add it to an existing plot.

For its list of Data Objects, the Data Manager displays several pieces of information. These are described below:

Name

  The name of the data object, unique among the set of data objects with the same type.

Type

  The type of the data object determines how it is created and what its options are. Type can be one of: Data Vector, Curve, Equation, Histogram, Spectrum, Plugin, Event Monitor, Matrix, Image, or Spectrogram.

Used

  This field gives the status of the data object. If there is a check mark in the Used column of a data object, then some other object in &kst; is dependent on it.

Samples

  The number of elements in the data object.

Properties

  A summary of the data object's key parameters, dependent on its type.

If you want to quickly find a vector among many items in the data manager, you may just type part of its name in the search field. Then, the data manager will respond by listing only those items which contain the entered text within their names.

### Creating and Deleting Data Objects

You can create new Data Objects using either the left panel in the Data Manager, or the Data menu.

When you are creating a new data object, you may enter a unique name to identify the object. If you do not enter a custom name then a unique name will be automatically generated.

To delete a data object you must use the Data Manager. Note that if the Used column for a data object has a check mark then some other data or view object in &kst; has a dependency on it. Depending on the strength of these dependencies, &kst; will prompt before an object is deleted. If, for example, other objects have critical dependencies on the one which you are attempting to delete, &kst; will ask if you would like to delete these other objects as well. Some dependencies are listed below:

- Plots are not critically dependent on the plottable objects which they contain, so if a plotted object is deleted &kst; will automatically remove it from all plots, without prompting.

- All data objects which use a particular data vector must be deleted before the data vector itself can be deleted.

- All children of a parent data object must be unused before the parent data object can be deleted.

After a sequence of deletions and removals of plottable data objects from plots, you may find that there are numerous unused data objects displayed in the Data Manager. To quickly remove these objects, you can click the Purge button.

## Data Types

### Vectors

Vectors are one of the most often-used data objects in &kst;. As their name implies, vectors are simply ordered lists of numbers. Usually they contain the x or y coordinates of a set of data points.

As vectors can potentially be quite large, it is a good idea to be aware of the amount of memory &kst; has available for use. The current available memory is displayed in the lower right corner of the status bar of the main &kst; window. If the status bar is not available, select Settings Show Statusbar to display it.

There are two kinds of vector — data vectors and static vectors. Data vectors are read into &kst; through a data source. Static vectors, on the other hand, are generated by &kst;. They represent an evenly spaced list of numbers, and can be useful, for example, for generating an x-axis for a plot.

#### Data Vectors

The following is a screenshot of the window displayed when editing data vectors. A new data vector is created if you choose the Read from data source radio button in the New Vector dialog. Explanations of the settings are listed below.

Edit Vectors
Edit Vectors

File name

  The path to the desired data file. Clicking the button to the right displays a file browser window that can be used to graphically browse for the file.

Field or column

  The field or column to use for the vector.

Start frame, Count from end, Range, and Read to end

  This section specifies the range of data to use from the selected field for the data vector. This discussion assumes a knowledge of the Frames concept. Using these four settings, the lower and upper boundaries of the data range can be set. For example, to read from frame 10 to frame 900, enter `10` for Start frame and `890` for Range. To read from frame 500 to the end of the file, enter `500` for Start frame and select the Read to end option. To read only the last 450 frames from the file, select the Count from end option and enter `450` for Range. The combinations used in the previous two examples are often useful for reading data from a file that is being updated in real time. Subsequent additions to the file are read, causing associated vectors to be updated as well.

Read 1 sample per and Boxcar filter first

  In addition to the lower and upper range boundaries, the samples to read from the selected range can be set. If Read 1 sample per is not selected, all samples in the selected range will be read. If Read 1 sample per is selected, only 1 sample per `N` frames will be read, where `N` is the number entered in selection box to the right. The value of the 1 sample that is used to represent a frame depends on whether or not Boxcar filter first is selected. If Boxcar filter first is not selected, the value is the same as the value of the 1st sample in the frame. If Boxcar filter first is selected, the value is the average of all the samples in that particular frame.

#### Static Vectors

The following is a screenshot of the window displayed when editing static vectors. A new static vector is created if you choose the Generate radio button in the New Vector dialog. Explanations of the settings for static vectors are listed below.

Edit Static Vectors
Edit Vectors

The entries of the vector will be From + (To - From)*(i-1)/(Number of samples - 1) for i = 1,...,Number of samples.

#### Generating Vectors from KstScript

You can also create and initialize vectors by using KstScript. See details in the KstScript of the Extensions Chapter, and in the Vector and DataVector Classes of the Script Chapter.

Examples are provided below:

##### Example of Using KstScript to Construct a Static Vector

    v = new Vector(); //construct a new vector
    v.tagName = "v1"; //give it a name in the Data Manager
    v.resize(100); //give it a length of 100
    for(var i=0; i<100; i++){v[i]=Math.random()};//intialize this vector

##### Example of Using KstScript to Load a Vector from Source Files

    source = new DataSource("rand.dat");//specify a data source file system. See the DataSource Class for details
    v = new DataVector(s,"1");//construct a new vector v and set its value from field 1 of the source file
    v.tagName = "vector" //give the vector a name in the Data Manager

### Curves

Curves are primarily used to create plottable objects from vectors. Curves are created from two vectors - an X axis vector and a Y axis vector, that provide a set of data points. Thus, a curve can be thought of as a set of data points and the lines that connect them (even though the points or lines may not be visible on plots).

The following is a screenshot of the window displayed when creating or editing curves. Explanations of the settings are listed below.

Edit Curves
Edit Curves

Legend Text

  The string to be used in Plot Legends to describe this curve.

X axis vector

  The vector to use for the independent (horizontal) axis.

Y axis vector

  The vector to use for the dependent (vertical) axis.

+X error bar and +Y error bar

  The vectors containing error values corresponding to the X axis vector and Y axis vector, respectively. The vectors should contain the sizes of the error bars in the same order as the data points.

Use +X/Y error bar for -X/Y

  If this item is checked, the error bars are drawn symmetrically about the data point. To draw asymmetric error bars in the x or y direction, uncheck the box, and choose a vector for the length of the error bar below the data point in the -X/Y error bar combo box.

-X/Y error bar

  The vector to use for the error bar below the data point. See the previous entry for more detail on using asymmetric error bars.

Color

  Determines the color of the data points of the curve. Clicking the button displays the standard &kde; color chooser.

Show points

  Determines whether or not the data points will be indicated by the selected type of marker.

Density

  If Show points is enabled, Density allows you to have &kst; show only occasional points if they would otherwise clutter the plot.

Show lines

  Determine whether or not the data points will be connected by lines

Weight

  If Show lines is enabled, this gives the thickness in pixels of the curve.

Bargraph

  Draws lines or rectangles from the data points to the y = 0 line in the plot.

Ignore in automatic axes range calculations

  If this option is checked, the curve will be ignored in determining the automatic y-axis range; if it is unchecked the curve will be used in determining the automatic y-axis range. By default, this option is unchecked.

Y axis vector offset

  If this option is checked, the corresponding selected scalar will be subtracted from the y values of the curve.

#### Using the Curves Class in KstScript

Plotting a curve can be done by using KstScript. See details in the Curve Class of the Script chapter.

##### Example of Using KstScript to plot a curve

    //xv and yv are names of two existing vectors in the Data Manager.
    //See example 3.1 and 3.2 for constructing vectors using KstScript.
    c = new Curve("xv", "yv") // to construct a new curve using xv and yv
    c.tagName = "xv vs yv"; //define the name for the curve in the data manager
     //to view the curve, a window object and a plot object need to be constructed.
    w = new Window();
    p = new Plot(w);
    // using the append method in the CurveCollection Class to show the curve in plot
    p.cuves.append(c);

There is a detailed example of vectors, plots and curves in the Extensions Chapter. Click here to view it.

### Equations

An equation data object consists of a mathematical expression and an independent variable. The expression is built using a combination of scalars, vectors, and operators, and usually represents the values of the dependent variable. The independent variable is generated from an existing vector, and stored as a slave vector along with the evaluated results of the equation.

The following is a screenshot of the window displayed when creating or editing equations. Explanations of the settings are listed below.

Edit Equation

Operators

  A list of available operators. Choosing an operator from the list inserts the operator at the current insertion point in the Equation text box.

Vectors

  A list of the current vector objects in &kst;. Choosing a vector from the list inserts the vector at the current insertion point in the Equation text box.

Scalars

  A list of the available scalar values. This list is composed of both the scalar values in the current &kst; session as well as some built-in constants. Choosing a scalar from the list inserts the scalar at the current insertion point in the Equation text box.

Equation

  The mathematical expression representing the independent variable. You may manually type in this text box or you may select items to insert using the above drop-down lists. When manually typing equation expressions, remember to put vector and scalar names into square brackets

  For a full list of the valid Kst operators, functions and constants to express equations, see the Equation Expression in Kst.

X Vector

  Select a vector to use as the independent vector. Select a vector from the drop-down list, or quickly create a new one by clicking the wizard button to the right.

Interpolate to highest resolution vector

  Select this option to interpolate the selected X vector to the greatest number of samples possible, given the data used in the equation expression. For example, if the data expression acts on a vector Y which contains twice as many variables as the selected X vector, the equation object will create a slave vector of interpolated X values with the same number of points as Y.

#### Using the Equation Class in KstScript

The Equation Class in KstScript provides methods to manipulate Equation objects. An example is shown below.

##### Example of Using the Equation Class

    //construct a vector object
    x = new Vector();
    x.tagName = "x_vector";//call it x_vector in the Data Manager
    x.resize(100);

    //initialize x_vector
    for(var i=0; i<100; i++){x[i]=i;}

    //construct an equation object
    eq = new Equation("LN([x_vector])",x);//specify the equation expressions
                                          //check equation appendix to see
                                          //what expressions are supported by &kst;.
    eq.tagName = "Natural Logarithmic Function";

    //to check if the equation is valid
    eq.valid
    //this should return true

    //to construct a curve using the resulting vectors of applying this equation
    c = new Curve(eq.xVector, eq.yVector);
    c.tagName = "eq_curve";
    //to view the curve, see Example 3.3.

### Histograms

This object calculates the histogram of a chosen vector.

The following is a screenshot of the histogram specific options when creating or editing histograms. Explanations of the settings are listed below.

Edit Histogram

Data vector

  The data vector to create the histogram from. Although a vector is needed to create a histogram, the vector is treated as an unordered set for the purposes of creating a histogram.

From and To

  The From field contains the left bound for the leftmost bin in the histogram. The To field contains the right bound for the rightmost bin in the histogram.

Num bins

  Enter the total number of bins to use in this field.

Auto Bin

  Instead of specifying values for From, To, and Num bins, you can click Auto Bin to automatically generate values for all three fields based on the length and lowest/highest values of the chosen data vector.

Number in bin

  Selecting this option causes the y axis to represent the number of elements in each bin.

Fraction in bin

  Selecting this option causes the y axis to represent the fraction of elements (out of the total number of elements) in each bin.

Percent in bin

  Selecting this option causes the y axis to represent the percentage of elements (out of the total number of elements) in each bin.

Peak bin = 1.0

  Selecting this option causes the y axis to represent the number of elements in each bin divided by the number of elements in the largest bin (the bin with the greatest number of elements).

#### Using the Histogram Class in KstScript

Histograms can also be constructed using KstScript. See details in the Histogram Class of the Working with KstScript appendix.

##### Example of Constructing a Histogram through KstScript

The example below uses the data file `normal.dat` generated by the perl script in Generating a Histogram of Chapter 2

    //load the data vector from normal.dat; see details in example 3.2
    s = new DataSource("normal.dat");
    v = new DataVector(s, "1");

    //construct a histogram object using vector v.
    hs = new Histogram(v);

    //construct a histogram curve; see details in the Curve Class
    c = new Curve(hs);

    //view the histogram; see example 3.3
    w = new Window();
    p = new Plot(w);
    p.curves.append(c);

### Spectrum

This data object calculates the Discrete Fourier Transform (DFT) of an input signal using a Fast Fourier Transform (FFT) algorithm, and outputs a spectrum of the signal.

The mathematical definition of the DFT is:

DFT

where `x(n)` is the input signal, and `N` is the number of samples.

The following definitions assume basic knowledge of power spectra — we use the conventions and terminology of Numerical Recipes in C: The Art of Scientific Computing, published by Cambridge University Press.

Power Spectra Window
Power Spectra Window

Data vector

  The data vector to create the spectrum from.

Apodize, Function, and Sigma

  An Apodization Function is a function used to reduce leakage.

  In the drop-down menu of Function, the apodization function applied to the spectrum can be chosen from:

  - Default

  - Bartlett

  - Blackman

  - Connes

  - Cosins

  - Gaussian (custom sigma)

  - Hamming

  - Hann

  - Welch

  If the Apodize option is selected, the data is windowed using the function chosen from the drop-down menu at right. Apodization can be used to reduce bin-to-bin leakage but the resolution of the plot will be decreased. If the Gaussian apodization function is chosen, then a sigma value may also be entered to define its effective width.

Remove Mean

  Select this option to remove the mean from the data while calculating the transform.

Interleaved average and FFT Length

  Selecting Interleaved average calculates the FFT over segments of the data, which increases the FFT speed and improves accuracy, at the result of less information about low frequency components. The length of the interleaved segments is specified as a power of 2. If Interleaved average is unchecked, &kst; will choose the smallest possible FFT length which is greater than the data vector length.

Interpolate Over Holes

  If the data vector contains NaN values, then selecting this option will cause &kst; to interpolate through them when calculating the power spectrum.

Sample rate

  This is the number of data vector samples per unit time. The sample rate is used to generate the X axis of the spectrum for plotting.

Vector units and Rate units

  The units specified in these textboxes are used for the purpose of auto-generating axes labels for the plots.

Output

  The output spectrum can be chosen from

  - Amplitude spectral density (V/Hz^1/2)

  - Power spectral density (V^2/Hz)

  - Amplitude spectrum (V)

  - Power spectrum (V^2)

  The option selected from this drop-down menu determines which variant of the power spectrum should be computed.

  The power spectrum is the plot of |X(k)|^2 vs frequency vector generated by the sample rate. The amplitude spectrum is calculated by taking the square root of the power spectrum. Spectral densities are calculated by multiplying the Power or Amplitude Spectra by the factor: N/sample rate, where N is the number of data samples.

#### Using the PowerSpectrum Class in Kst

The PowerSpectrum Class in KstScript can also be used to construct power spectra.

##### Example of Using PowerSpectrum Class

    //construct a random vector whose value is between 0 and 1 with 10000 samples
    x = new Vector();
    x.resize(10000);
    x.tagName = "random" //give it a name: random in Data Manager
    for(var i=0; i <10000; i++){x[i]=Math.random()}

    //construct a Power Spectrum according to the constructor:
    //PowerSpectrum(vector, freq, [,average[,len[,apodize[,removeMean]]])
    ps = new PowerSpectrum(x, 10, true, 10, true, false)
    ps.tagName ="rand_Spec";

After executing the above scripts, the Data Manager should look like this:

resulting Names in DataManager
resulting Names in DataManager

Later, the properties of this power spectrum can be changed like this:

    ps.removeMean = true; //choose the remove mean option
    ps.length = 13; //change the FFT length to be 2^13

### Spectrogram

A spectrogram is a way to plot the frequency spectrum of a data vector as a function of position within the data. It allows you to visualize, for example, how the frequency distribution of a signal changes with time. In &kst; the spectrogram data object takes a data vector as an input and produces an image object which can be displayed as a waterfall plot. The waterfall plot is used to show how two-dimensional information changes over time; here, it indicates how power spectrum vs frequency of a data set changes over time. An example is shown in the screenshot below. This is a spectrogram from one day of data from a channel of the WMAP satellite. The size of the intervals over which the component spectra are calculated has been chosen so that each interval corresponds to 1/2 hour of data. The 1h periodic precession of the satellite about its spin axis can be clearly seen. Also, at ~0.007Hz the red lines at the bottom of the image show the much faster (0.5rpm) spin frequency of the satellite.

Spectrogram Example

Here is the edit dialog for the spectrogram object.

Edit Spectrogram Window

An explanation of the spectrogram specific options is given below:

Window Size

  The length of the data over which to take the spectra. The data vector will be subdivided into intervals of this length, and a spectrum will be computed for each one.

FFT Options

  These are the options for the individual spectra. See the Spectrum data object for more information. The output quantities are indicated by color scalar.

Color Palette

  This option should be accessed from the Edit Image dialog of the image created from the spectrogram.

  Here you can choose the color scheme to use for the spectrogram. The amplitudes of the spectra are color-coded according to the selected scheme.

#### Using the Spectrogram Class in KstScript

The spectrogram can also be generated by using the Spectrogram class in KstScript

##### Example of Using the Spectrogram Class

The vector: random generated in Example 3.6 will continue to be used in this example

    s = new Spectrogram("random", 10)//generate a spectrogram using vector random
                                   //and specify the frequency to be 10 Hz
    s.windowsize = 500; // specify the window size to be 500
    s.tagName = "rand_Spectrogram";

Now, you can check that rand_Spectrogram is in the Data Manager. You can view the waterfall plot by making an image from the slave matrix

### Plugins

A plugin data object represents a &kst; plugin. All plugins have a common format, and show up as type Plugin in the Data Manager. For more information about the available plugins and their options, please see the Plugins chapter.

### Event Monitors

An event monitor data object essentially keeps track of one or more vectors or scalars, and performs an action when a specified condition involving the vectors and scalars is true. Event monitors are usually used in conjunction with live, or streaming data. For example, an event monitor could be created to monitor whether or not elements of a data vector representing temperature exceed a predefined value.

You can create a new event monitor from the Data New Event Monitor..., or by using the Data Manager.

The following is a screenshot of the window displayed when creating or editing events. Explanations of the settings are given below.

Edit Event Monitor

Expression

  The expression to monitor. You may type directly in this textbox, or select items to insert using the operator, vector, and scalar drop-down lists. Ensure that the expression entered in this textbox is a boolean expression (i.e. it evaluates to either true or false). This usually entails using an equality or inequality in the expression. Note that vectors entered in the textbox will be monitored according to their individual elements.

  Whenever this expression is true (evaluates to non-zero), the event will be triggered. The action taken upon an event trigger depends on the settings specified in the other settings.

Description

  This textbox is used to store a short description of the event. The description is primarily available as a method for keeping track of multiple events. You can enter any text you wish in this textbox.

Log as:

  Enable this checkbox to have entries added to the &kst; debug log when this event is triggered. There are three types of entries in the debug log — notices, warnings, and errors. Select the desired type of entry using the corresponding radio button.

E-Mail Notify

  Enable this checkbox to have &kst; send e-mail notifications to the specified address when this event is triggered. General E-mail configuration options are available in the &kst; Settings dialog.

ELOG

  If the ELOG Extension is activated, then then Event Monitor can provide notification through ELOG. This can be useful for remotely monitoring live data, or for obtaining summaries of event activity. Please see the ELOG section for more information.

KstScript

  If this option is activated, then the Event Monitor can execute a KstScript when it triggers. Please see the KstScript documentation for more information.

### Matrix

A matrix represents a set of three-dimensional data points (x, y, z) arranged in a two-dimensional grid.

Matrix Structure

Just as is the case with the vector datatype, there are two kinds of matrix: data and static. Data matrices are read in from data sources, and static matrices are generated by &kst;.

#### Data Matrix

The following is a screenshot of the window displayed when editing a data matrix. A new data matrix is created if you choose the Read from data source radio button in the New Matrix dialog.

Edit Data Matrix

All of the options which are available for data matrices are analogous to the options for Data Vectors.

Data Source Parameters

  Any field in ascii file that is going to be used as data source for matrix must have its field name in this format: [MATRIX, # of rows, x min, y min, x step size, y step size]. A very simple example of a matrix source file is provided below.

Data Range

  This option allows you to use only a portion of the matrix by specifying the range of data in both the x and y directions.

  - X starting frame: the starting index in x dimension of the matrix

  - Y starting frame: the starting index in y dimension of the matrix

  - X number of frames: the number of data points in x dimension

  - Y number of frames: the number of data points in y dimension

  The simple example below shows how to use this option.

##### Example of a Simple Matrix Source File

    [MATRIX,5,0,0,1,1]
    0.47
    0.12
    0.49
    0.97
    0.67
    0.46
    0.64
    0.9
    0.53
    0.13

The resulting matrix would be a 2×5 matrix, and the 10 samples are all included: simple_matrix simple_matrix

Notice that only the number of rows needs to be specified since Kst can figure out the number of columns automatically in order to have the loaded matrix use the largest number of samples in the original data field.

If the field name of the matrix is [MATRIX,6,0,0,1,1], the resulting matrix would have dimension 1×6; only 6 samples would be included.

##### Example of Specifying a portion of matrix

For the 2×5 matrix in the Screenshot above, if only part of the matrix:from index(1,2) to index(2,4), is needed; that is

0.12 0.64

0.49 0.9

0.97 0.53

The Data Range needs to be specified in the following way:

- X starting frame: 0

- Y starting frame: 1

- X number of frames: 2

- Y number of frames: 3

The result is this:

extract-matrix
extract-matrix

#### Static Matrix

The following is a screenshot of the window displayed when editing static matrices. Static matrices allow you to create gradient matrices such as the one shown in the background of the annotation object example. A new static matrix is created if you choose the Generate gradient radio button in the New Matrix dialog.

Edit Static Matrix

These options allow you to construct a matrix which has a linear gradient in Z along either the X or Y axis.

The steps options allow you to set the size of the matrix.

- X steps indicates column numbers

- Y steps indicates row numbers

The X/Y minimum and X/Y step size options allow you to set the X and Y ranges.

#### Using Matrices through KstScript

Matrices can also be constructed by KstScript. See details in the Matrix class and the DataMatrix Class of the Script Chapter.

##### Example of Using the DataMatrix Class in KstScript

    s = new DataSource("home/vyiwen/mx.dat")//mx.dat is the same matrix source file in the example source file above
    m = new DataMatrix(s, "[MATRIX,5,0,0,1,1]")//use mx.dat to construct a matrix
    m.tagName = "Simple_matrix";//Name the matrix Simple_matrix and you should be able to see it in the Data Manager.

    //if you want to specify a portion of matrix by specifying the data range in the x, y dimension,
    //use the method: change(xStart, yStart, xFrames, yFrames)

    m.change(0,1,2,3)//you can see the change in Simple_matrix
                     // it should be the same 2 by 3 matrix
                     // as in the example of specifying a portion of matrix above.

##### Example of Using the Matrix Class to construct a Static Matrix

    y = new Matrix(); // to construct a new matrix
    y.resize(3,5);  //define the size of the matrix 3 by 5 matrix
                    //i.e 3 rows and 5 columns
    for(var i=0; i<5;i++){for(var j=0; j<3;j++){y.setValue(i,j,i+j)}}; //to initialize the matrix
    y.tagName="mx"; //you should see matrix mx in the Data Manager now

### Images

Images are graphical representations of Matrices. Images can be plotted as color maps, contour maps, or both.

Image Window
Image Window

Matrix Selection

  Select the matrix to use for this image. New matrices can be created or edited by clicking buttons to the right.

Color Map, Contour Map, and Color Map and Contour Map

  Select the type of image to be plotted. Changing this selection enables or disables sections of the image dialog as appropriate. A color map represents the Z value of each cell in the matrix using a color. A contour map plots curves which follow lines of constant Z.

Color palette

  Select the color palette to use for the color map.

Threshold - Lower and Upper

  Enter the lower and upper thresholds to use for the mapping of colors. Palette colors are evenly distributed between the lower and upper thresholds. Any cells in the selected matrix with z values less than the lower threshold are mapped to the first color in the palette. Any cells in the selected matrix with z values greater than the upper threshold are mapped to the last color in the palette.

Max/Min

  Clicking this button causes the lower and upper threshold values to be set to the current minimum and maximum z values of the selected matrix.

Smart and Percentile

  Clicking the Smart button causes the lower and upper threshold values to be set such that the percentage of z values in the selected matrix contained between them is equal to the value in the Percentile numberbox.

Real-time auto threshold

  Selecting this option causes the lower and upper threshold values to be always set to the minimum and maximum of the z values for the selected matrix, even when the matrix updates.

Number of contour levels

  Select the number of contour levels to use. The contour levels will be distributed evenly between the lowest and highest z values found in the matrix.

Color

  Select the color to use for the contour lines. Clicking this button displays a standard &kde; color chooser dialog.

Weight and Use variable line weight

  Select the Weight, or thickness of the contour lines. If Use variable line weight is selected, contour lines representing higher elevations are drawn thicker than those representing lower elevations. Use discretion when selecting this option, as images with high contour line densities may become unreadable.

#### The Image Class in KstScript

See the Image Class of the Script Chapter for detail.

### Vector View

Vector view provides a way to view a portion of the data plot. It excludes any data point which is not in the range of the X/Y limits specified by the user.

Edit Vector View
Edit Vector View

Input Vectors

  Select an input X/Y axis vector from existing vectors or create one by using the shortcut icons.

View Ranges

  Adjust the view range of the plot by selecting X min/Y min and X max/Y max values for the input vectors.

Flag Vector

  This option is used to exclude specific points in the input vector for the vector view. The indices of non-zero points in the flag vector indicate that the data points of the input vector with the same indices should be filtered out for its vector view.

## Data Tools

### The Data Wizard

The Data Wizard provides a graphical quick and easy way of importing data into &kst;- automatically creating vectors, curves, power spectra, and plots. To launch the wizard, select Tools Data Wizard or click on the icon data wizard icon in the toolbar. The Data Source pane will appear as following and you will need to select a data source file.

data source
data source

#### Configuring Data Source

After selecting the data source file, you may need to configure the file by clicking on the Configure... button. For information on how to use the Configure Data Source dialog, see the ASCII datasource section.

After configuring the data source file, click on the OK button to close the Configure Data Source dialog, and click on the Next button in the Data Source pane. The Select Data pane will appear as shown below.

#### Selecting Data

Data Wizard Select Data Dialog
Data Wizard Select Data Dialog

Here you can select the fields you wish to import into &kst;. The right and left arrow buttons allow you to move data vectors back and forth between the Available data and the Selected data lists. The up and down arrow buttons allow you to move the positions of the selected vectors up and down in the Selected data list. You may filter the list of fields by entering a string to match (wildcards such as `*` are supported) into the text box above the list. Click the Next button to advance to the Data Presentation pane.

#### Presentation

Data Wizard Data Presentation Pane
Data Wizard Data Presentation Pane

The Data Range section is used to specify the range of data to read from the selected vectors in the input file. For information about these parameters, read the description of the Vector data object.

Power Spectrum and X axis settings can be specified within the Plot Types section. These settings are described below.

XY, Power Spectrum, and XY and Power Spectrum

  Select whether to plot the data (XY) only, Power spectrum only, or both. If the power spectrum is selected for plotting, additional settings in this section become available.

X Axis Vector

  The vector to be used as the independent vector for the plots. You may select a field from your data file, or the INDEX vector. The INDEX vector is simply a vector containing elements from 0 to N-1, where N is the number of frames in the data file.

The FFT Options subsection in the Plot Types section is available only if a power spectrum is to be plotted. Detailed information on these options can be found in the Spectrum section.

Once you are satisfied with all the settings, click the Next button to advance to the final window.

#### Configure Plot Layout

Data Wizard Configure Plot Layout
Data Wizard Configure Plot Layout

From here, you can change general plot settings. Most of the settings are self-explanatory. A brief overview of each section is provided below.

Curve Style

  Select whether to plot data points only, lines connecting the data points only, or both. By default, lines are continuous with weight 0, and data points are marked using crosses. Line and data point colors are chosen so that curves with identical colors are minimized. Note that the curve style settings apply to both Power Spectra and XY plots.

Curve Placement

  Select the plots to place the new curves on. Cycle through distributes the curves on the plots by cycling through the plots.

  For example, if you have 4 curves, and you choose to cycle through 2 plots. The result will be that curve 1 is placed in plot 1; curve 2 is in plot 2, and this is the first plot cycle. Curve 3 is back to plot 1; curve 4 goes to plot 2, and this is the second cycle. Note that the curve placement settings apply to both Power Spectra and XY plots.

Label Generation

  Select the desired labels and legends to be placed on the plots.

Plot Placement

  Select the desired window(s) to place the new plots in. New windows can be created for the plots by selecting In new window.

Power Spectrum Axis Mode

  Select these check-boxes if you would like the automatically created power spectra to have logarithmic X or Y axes.

Once you are satisfied with all the settings, click the Finish button and the plots will be generated.

### The Data Menu

The Data menu provides quick access to many features related to data objects in &kst;. Most of the menu functions duplicate functions found elsewhere, so only brief descriptions are provided below.

Reload

  Reloads all data vectors from their source files.

Data Manager

  Displays the Data Manager.

View Manager

  Displays the View Manager.

New [data object]

  Displays the corresponding dialog for creating the data object. Refer to Data Types for descriptions of each dialog.

View Scalar Values

  Displays a dialog from which the values of all the scalars in the current &kst; session can be viewed. The values are displayed hierarchically, based on which higher level data objects the scalars are determined for. The dialog is updated dynamically if the values change.

  View Scalar Values
View Vector Values

  Displays a dialog from which the values in all the current vectors can be viewed. Select a vector to view using the drop-down list. The dialog is updated dynamically if the values change.

  View Vector Values
View Matrix Values

  Displays a dialog from which the values in all the current matrices can be viewed. Select a matrix to view using the drop-down list. The dialog is updated dynamically if the values change.

  View Vector Values
View Fit Results

  Displays a dialog which shows all the resulting values from fit plugins. Select a fit result to view using the drop-down list. The dialog is updated dynamically if the values change.

  Curve Fitting - Fit Results

### Data View Mode

Data mode allows precise viewing of the data points used in a plotted data object. To toggle data mode, select Plots Data Mode, or click the Data Mode icon on the Toolbar. Now, when the cursor is moved over a plot, a colored dot will indicate the closest data point to the cursor, as shown in the screenshot below. The status bar will display the coordinates of the data point (in terms of the x and y vectors used to plot the data object) in the status bar at the lower left corner of the &kst; window. The status bar will also display the x, y, and z coordinates of any visible image. If images overlap, only the coordinates of the topmost image will be displayed. Note that all zooming functions are still available while in data mode.

Data Mode

If the status bar is not visible, select Settings Show Statusbar

### Change Data File

One common mode of using &kst; is to create a session of plots and analysis for a particular type of data. Then, the same plots and analysis can be performed easily on another set of data simply by changing the data source file, provided that the two data files are similar. For example, for ASCII data files, similar data files need to have same number of fields.

The Change Data File dialog can be accessed by choosing Tools Change Data File or clicking on the icon icon changedatafile in the toolbar. The dialog which appears is shown below:

To use this dialog, select the vectors which you would like to change the data file of using the list in the middle of the dialog. The All From, Clear, Select All, and search field can help you to do this. The use of wildcards is supported in the search field.

Then, select a new data file and press Configure button to configure data source. If the Change selected vectors and matrices radio button is selected, then when you press the Apply or OK button, the plots of the selected data vectors will be replaced by the data vectors loaded from the same fields of the new data file that you specified.

If the Duplicate selected vectors and matrices radio button is selected, vectors coming from the new file will be plotted together with the existing vectors in exactly the same way.

If some of the selected vectors have dependents (curves, spectra, histograms, etc.) then the Duplicate dependents option allows you to duplicate these objects as well.

### Change Data Sample Ranges

The Change Data Sample Ranges dialog can be accessed through the Tools menu or the icon icon_datarange in the toolbar. It allows you to modify how a set of vectors are read in from their associated files. The options are detailed in the Data Range option of the Data Vectors documentation.

### Common Icons

As mentioned above, most data tools have corresponding icons in the toolbar. There are also some useful icons commonly seen in various dialogs.

create vector icon

  This icon opens the New Vector dialog, the same as choosing Data New Vector...

edit vector icon

  This icon opens the Edit Vector dialog where you can edit the properties of the currently selected vector.

create mx

  This icon opens the New Matrix dialog, the same as choosing Data New Matrix...

edit matrix icon

  This icon opens the Edit Matrix dialog where you can edit the properties of the currently selected matrix.

clear icon

  This icon is commonly seen in the search field of many dialogs used to list data objects (vectors, curves, etc). For example, it appears in the Data Manager, View Scalar Values, and Change Data File, etc.

  It is used to quickly clear the textbox of the Search field and restore the original list of data objects.

create scalar icon

  This icon opens the New Scalar dialog as shown below: new scalar new scalar

  Here, you can define your own scalar by typing its name and associated value.

edit scalar icon

  This icon is used to change values of user-defined scalars. Essentially, it opens the same dialog as shown above.

## Datasources

&kst; uses externally linked datasources to read from data files. This means that new data formats can be transparently added to the &kst; repertoire.

Currently, the &kst; distribution includes the following default datasources:

- ASCII

- Dirfiles

- HEALPix

- QImage

- Indirect

- CDF

- netCDF

- LFIIO

- SCUBA

If you have a data format which you would like &kst; to read, it's easy to create your own.

All of the data sources read from files using KIO, which is usually able to load data through common file compression formats (e.g. BZIP2, GZIP, ZIP, TAR) and most popular networking protocols (e.g. HTTP, FTP, SFTP, SMB). For more information on which formats are supported on your system type ``kinfocenter`` at the command line and inspect the protocols tab. If &kst; detects that you have typed a URL into the filename of a vector or matrix, for example, a Connect button will appear. Clicking the Connect button instructs &kst; to load the indicated data, and channel it to a suitable data source.

The following concepts are important in understanding how &kst; works with data sources. Some terminology is also introduced in this section.

Samples

  A sample is considered to be the fundamental unit with respect to data files. Each sample consists of one data value in the file. Note, however, that one sample may not correspond to one value in a data vector in &kst;, due to the concept of frames.

Fields

  A field usually corresponds to one vector in &kst;. For example, a column in an ASCII data file is considered to be one field. Each field can have an explicit or implicit name. Datasource readers provide functions for reading and obtaining fields and field names.

Frames

  A frame corresponds to a set number of samples, and each field in a data file can have its own ratio of samples per frame. The size of a data file is measured by the number of frames it contains. For ASCII data files, the number of samples per frame is 1, but for some data files, there may be multiple samples per frame. In the below illustration, the first 3 frames of an imaginary data file are shown. In this particular data file, Field 1 has a ratio of 5 samples per frame, Field 2 has a ratio of 2 samples per frame, Field 3 has a ratio of 3 samples per frame, and Field 4 has a ratio of 1 sample per frame.

  Depending on the specific data vector settings in &kst;, data from files may be read as frames instead of samples, with either the first sample in a frame or the mean of all the samples in a frame representing the value of the frame. This capability can be useful for reducing extremely large data-sets to a more manageable size.

INDEX field

  Some data files may not have a field that represents the x axis of a plot. However, &kst; implicitly creates an INDEX field for all data sources. The INDEX field simply contains integers from 0 to N-1, where N is the number of frames in the data file.

### ASCII

The simplest input file format is the ASCII text file. These files are usually human-readable and can be easily created by hand or simple scripts if desired. The following is an example of an ASCII input file.

    112.5 3776 428
    187.5 5380 429
    262.5 5245 345
    337.5 2942 184
    412.5 1861 119
    487.5 2424 138
    567.5 2520 162
    637.5 1868 144
    712.5 1736 211
    787.5 1736 211
    862.5 2172 292
    937.5 1174 377
    1000.5 499 623

Datasources - Configure ASCII

Comment Indicators

  By default, commented lines in ASCII files start with one of the characters in this set `{#, !, /, ;, c}`. All commented lines and empty lines are ignored by &kst;. Valid numbers include those with decimals, negative signs, or `e`, to indicate scientific notation. Invalid numbers (such as English words) are replaced with 0 by &kst;.

Interpret ... As ...

  See the corresponding section for the Plot Dialog.

Always accept files matching:

  Here you can enter an expression which you want to designate ASCII data files, in case they are being claimed by another data source.

Data starts at line:

  Data before this line will be ignored. Setting this value to zero ensures that all of the data will be read.

Read field names from line:

  If this option is selected, the field names for the file's vectors will be read from the specified line. If the columns do not have labels, field names are assigned by &kst; based on the order of the columns (with the leftmost column having a field name of `1`).

Data format

  Choose the option which describes how the data is delimited (how the columns and rows are separated). Each column of this file represents a field, while each row represents one frame. Columns are typically separated by tabs or spaces, and rows are separated by carriage returns.

### Dirfiles

Dirfiles are a very simple data source designed to be read efficiently by &kst;. The data source is a directory which contains each raw field in its own binary file. Because the data for a given field are stored sequentially on disk in a binary format, reading is very efficient, typically limited by hard disk speed. Fixed and floating point binary formats are supported, as well as calibrations, bit fields and linear combinations of channels.

The directory contains one file for each field. These files contain only the raw samples, written sequentially. As in the rest of &kst;, each field in a datasource must have the same number of frames, but each field may have a different (fixed) number of samples per frame. Formatting and calibration information is contained in a file in the Dirfile called `format`, which lists each field and its properties. Below is example of such a file: In this file, a # specifies a comment, and for all fields, the first column specifies the field name.

    ##### Raw fields ####
    # data types:
    # c: 8 bit signed
    # u: 16 bit unsigned.  U: 32 bit unsigned
    # s: 16 bit signed     S: 32 bit signed
    # f: 32 bit float      d: 64 bit float
    # The fieldname is also the name of the raw binary file
    # holding the data.
    # Fieldname  RAW    datatype   samples/frame
    sensor1      RAW       s         1
    sensor2      RAW       U         20
    sensor3      RAW       c         1
    #
    #### derived fields ####
    # LINCOM fields: (F_out = F1_in*m+b [+ F2_in*m+b [+ ...]])
    # Fieldname  LINCOM  N  Field1_in       gain        offset
    S1V          LINCOM  1  sensor1    1.52587890625E-4 0.00
    #
    # LINTERP Fields: Calibrate using an ascii lookup table.
    # lookup table format:
    #     2 whitespace separated columns
    #     The first column is raw values, and the second is
    #     a corresponding calibrated value.  The file must be
    #     sorted by column 1.
    # The table is linearly interpolated to the value of
    # Field_in.
    # Fieldname  LINTERP  Field_in  Calibration_file
    S2K          LINTERP  sensor2   /data/etc/calibration/C2K.cal
    #
    # BIT values: the value of 1 bit of raw data: 0 or 1
    # Fieldname  BIT  Field_in  bit_num (0 to 7 for chars)
    RELAY1       BIT  sensor3   0
    RELAY2       BIT  sensor3   1
    #
    # The next line includes the contents of another format file
    # into this one.  Any fields referred to in this file will be
    # looked for in ../work/, not in ./, so this is essentially
    # including an entire other datasource.
    INCLUDE ../work/format

The following code fragment (which foolishly ignores all pretense at error checking) could be used to create this data source...

      char bits[1000];
      short s1[1000];
      unsigned int s2[20000];
      int fp;

      ......
      /* fill bits, s1, and s2 with data...*/
      ......

      fp = open("/data/example/sensor3",  O_WRONLY|O_CREAT, 00644);
      write(fp, bits, 1000*sizeof(char));
      close(fp);

      fp = open("/data/example/sensor2",  O_WRONLY|O_CREAT, 00644);
      write(fp, bits, 1000*20*sizeof(unsigned int));
      close(fp);

      fp = open("/data/example/sensor1",  O_WRONLY|O_CREAT, 00644);
      write(fp, bits, 1000*sizeof(short));
      close(fp);

      /* create the ascii file /data/example/format, listed above */
      /* create the cal file, /data/etc/calibration/C2K.cal,
         described above. */

      ......

Dirfiles can be used for real time data, but as each field is written separately, this requires some extra care. First of all, the writing application should avoid all buffering or caching - raw writes as in the above example work fine. Secondly, the order in which the fields are written need to be considered: &kst; determines the number of frames available in the data source from the size of the file containing the first RAW field encountered in the format file. So the first RAW field in `format` needs to be the last one written, as in the above example. If the data source is being read over NFS, attribute caching needs to be turned off.

When selecting a dirfile for use in &kst;, the directory containing the field files should be selected. &kst; will then automatically look for a `format` file, if it exists, to determine the fields and their properties.

### HEALPix FITS files

HEALPix is a pixelisation scheme for the sphere. More information can be found at http://healpix.jpl.nasa.gov/. HEALPix data is typically be loaded into &kst; as a matrix. For this data source to work you must have the CFITSIO libraries installed. An image of a sample HEALPix matrix is shown below:

HEALPix example

Here is the configuration dialog for the HEALPix data source, an explanation of the possible options is given underneath:

HEALPix example

Dimensions

  The size of the matrix to create based on the HEALPix data. The larger the matrix, the higher the resolution. If you choose dimensions which are larger than the resolution of the data in the HEALPix file, you will be able to see the shape of the HEALPix pixels in the image of the matrix.

Theta / Phi Range

  The colatitude / azimuth range of data to extract.

Theta / Phi Range

  The colatitude / azimuth range of data to extract.

### QImage

The QImage datasource allows you to read image data into &kst; matrices from a variety of formats. The collection of supported formats depends on the libraries installed on your system, but most popular formats are generally readable. If you'd just like to add a picutre to a plot, then the Picture annotation object is a better choice to use. An example of an image plotted from a QImage sourced matrix is shown below:

QImage example

### Indirect

This is a helper data source, which allows you to access data through a file pointer. If you put the URL or path of a data file on the first line of a text file with the extension `.cur`, then the Indirect data source is used to load the referenced file. The use of this is that the contents of the `.cur` file can then be externally modified to change the data file which is loaded by &kst;.

### CDF

The Common Data Format (CDF) was developed by NASA. It is described as a self-describing data format for the storage and manipulation of scalar and multidimensional data in a platform- and discipline-independent fashion.. This means that it is a generalized file format for storing and retrieving data, similar to FITS. For this datasource to work you must have the CDF libraries installed.

### netCDF

The network Common Data Form (netCDF) was developed by Unidata. They describe it as a set of interfaces for array-oriented data access and a freely-distributed collection of data access libraries for C, Fortran, C++, Java, and other languages. The netCDF libraries support a machine-independent format for representing scientific data. Together, the interfaces, libraries, and format support the creation, access, and sharing of scientific data.. In practice, it is similar to the FITS and CDF formats. For this datasource to work you must have the netCDF libraries installed.

### LFIIO

The LFIIO datasource reads FITS format data files created by the LFI component of the Planck satellite. It is currently the default reader for FITS files in &kst;. For this datasource to work you must have CFITSIO installed.

### WMAP Reader

This datasource allows you to load vectors from WMAP Time Ordered Data (TOD) files. For more information, see http://lambda.gsfc.nasa.gov/. For this datasource to work you must have CFITSIO installed.

### SCUBA File Reader

This datasource allows you to read the datafiles produced for SCUBA, an instrument on the JCMT telescope in Hawaii.