Python Hyperspectral Analysis Tool (PyHAT)

PyHAT is built on a simple .csv data format that is read into a Pandas data frame for maximum flexibility. Each row of the data frame contains one spectrum and associated metadata and compositional information if available. Each column uses a two-level labeling system which allows the user to do coarse (such as selecting all spectra) or fine (such as selecting the column corresponding to a single wavelength) data selection for simple access to the data.

PyHAT works with hyperspectral cubes by flattening them to match the tabular PyHAT format. Geographic information is stored as metadata so the cube can be reconstructed as needed. PyHAT has built-in support for reading and generating summary parameters from Moon Mineralogy Mapper (M3) and the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) data cubes. Other data formats can be used if flattened into the PyHAT format, and future work will add more support for common data sets.

PyHAT includes multiple baseline removal algorithms that can be used to remove non-informative signal and leave only the useful features of a spectrum. Built-in plotting functions allow the user to visualize how these algorithms are working and decide which is most suitable. 

PyHAT leverages the Scikit-Learn library for many capabilities., including outlier identification. Two algorithms for outlier identification are implemented: Local Outlier Factor (LOF) and Isolation Forest (IF). Once spectra have been flagged as potential outliers, data manipulation functions allow them to be removed or partitioned into a separate data set.

A key step in the process of training a regression model is tuning its hyperparameters. To avoid overfitting, models must be cross-validated by iteratively withholding some of the training data and predicting it as if it is unknown. This allows the user to identify the parameters that will result in the best balance between training set accuracy and generalizability. PyHAT provides a cross validation module that makes this process simple and utilizes parallel processing to run through the often time-consuming cross validation calculations more efficiently.

Once model parameters have been tuned, PyHAT provides the ability to perform predictions on novel data. Multiple different regression methods can be optimized and compared, and the plotting functions include settings to produce a "one-to-one" plot comparing predicted vs actual values to aid in model evaluation and comparison. PyHAT also includes the ability to blend the predictions from multiple submodels, so that the benefits of specialized models can be combined (see related publications for more details).