Microscopic Hyperspectral Image Analysis via Deep Learning

Hyperspectral imaging (HSI) is a technique that can obtain more spectral information than that in normal color images. Due to this property and strength in material classification, it is widely used in remote sensing, agriculture, and environmental monitoring. In recent years, with the rapid developments of hardware, hyperspectral cameras have become more portable and a ordable. An increasing number of studies are being conducted on HSI systems, and research focuses have expanded from remote sensing to close-range objects. With a proper microscopic kit, a hyperspectral camera can capture images of objects of micrometers in size. In this thesis, an HSI system is introduced which consists of a hyperspectral camera, a microscope, control software, and an image processing workstation. The samples are placed under the microscope which has the camera mounted on the top. The parameters of the camera can be tuned by the control software to have the best image quality. After the setup, the camera takes the HSI image of the samples. Then, the image is transferred to the workstation and saved as a raw HSI image for further process. Two datasets of cells and microplastics are collected and introduced as benchmark datasets for this research. The reason to build these two benchmarks is because of their demands. In the area of cell viability assay, traditional methods use uorescent dyes to distinguish live and dead cells. Although working very reliably, they require physical contact with the cells, which a ects the appearance of the cells and some of the original cell features. As a consequence, there is a demand for the development of non-invasive technology for cell analysis. Our HSI system is capable of using computer vision techniques to classify live and dead cells as a non-invasive and systematic method so that the property of the cells can remain unchanged and the system can be operated without special skills. The microplastics dataset is built to address the needs of environmental protection which is an important research topic with significant social and economic values. The increasing amount of microplastics in the ocean has attracted enormous concern because of its potential to damage the ecosystem and a ect the health of humans and animals. While HSI has shown great potential in analyzing microplastics, studies in this direction are hindered by the lack of public available image data. Therefore, there is an urgent so that there is an urgent demand to build a dataset for microplastics detection. After the datasets have been constructed, we evaluate the support vector machine (SVM) on them for the baseline approach. We apply several feature extraction methods to process the HSI images of the cells before feeding them into the SVM, including extended morphology profile (EMP), tensor morphology profile (TMP), 3D scale-invariant feature transform (SIFT3D), 3D local derivative pattern (3DLDP) and spectral-spatial scaleinvariant feature transform (SS-SIFT). Among them, TMP has the best performance for the cell classification task. Regarding the detection of microplastics, the spectral signature is used to extract the feature and is fed into SVM for detection. Furthermore, we propose a novel attention-based convolutional neural networks (CNN) to classify the cells to take advantage of the development in deep learning. Inspired by the VGG networks, we first build a classification network for our hyperspectral data. Then, a band weighting network and a spatial weighting network are integrated into the backbone. The band weighting network assigns a weight to each band in the hyperspectral images. The weights can suppress redundant bands that do not make an important contribution to the classification task and make the classification network focus on the bands that have more important features for classification. The spatial weighting network assigns a weight to each pixel in the hyperspectral images. The weights can help the classification network focus on important parts of the images and ignore the irrelevant parts. These two weighting networks help to improve the final classification accuracy of the cells. In the experiments on hand-crafted features, SVM with TMP feature extraction method has the best accuracy of 83.72% for the cell classification task. SVM with spectral signature produces 99.13% accuracy on the microplastics detection task. In comparison, the attention-based CNN achieves 98.17% for the cell classification task. These results show that our HSI system and classification methods have great potential for these two classification and detection tasks. The richness of spectral information that is provided by hyperspectral images has a great potential in material recognition tasks, helping to classify di erent materials based on their unique spectral signatures of each material. Because of this, our research can contribute to a wider range of biomedical and environmental domains.