CAIR Deep Learning Research Publications

2026

Mouton, C., Rabe, R., Koser, N. C., Krekiehn, N., Hansen, C., Hövener, J.-B., & Glüer, C.-C. (2026). Useful nonrobust features are ubiquitous in biomedical images. In . IEEE International Symposium on Biomedical Imaging (ISBI).

We study whether deep networks for medical imaging learn useful nonrobust features - predictive input patterns that are not human interpretable and highly susceptible to small adversarial perturbations - and how these features impact test performance. We show that models trained only on nonrobust features achieve well-above-chance accuracy across five MedMNIST classification tasks, confirming their predictive value in-distribution. Conversely, adversarially trained models that primarily rely on robust features sacrifice indistribution accuracy but yield markedly better performance under controlled distribution shifts (MedMNIST-C). Overall, nonrobust features boost standard accuracy yet degrade outof-distribution performance, revealing a practical robustnessaccuracy trade-off in medical imaging classification tasks that should be tailored to the requirements of the deployment setting.

@inbook{613,
  author = {Coenraad Mouton and Randle Rabe and Niklas Koser and Nicolai Krekiehn and Christopher Hansen and Jan-Bernd Hövener and Claus-C. Glüer},
  title = {Useful nonrobust features are ubiquitous in biomedical images},
  abstract = {We study whether deep networks for medical imaging learn useful nonrobust features - predictive input patterns that are not human interpretable and highly susceptible to small adversarial perturbations - and how these features impact test performance. We show that models trained only on nonrobust features achieve well-above-chance accuracy across five MedMNIST classification tasks, confirming their predictive value in-distribution. Conversely, adversarially trained models that primarily rely on robust features sacrifice indistribution accuracy but yield markedly better performance under controlled distribution shifts (MedMNIST-C). Overall, nonrobust features boost standard accuracy yet degrade outof-distribution performance, revealing a practical robustnessaccuracy trade-off in medical imaging classification tasks that should be tailored to the requirements of the deployment setting.},
  year = {2026},
  month = {04/2026},
  publisher = {IEEE International Symposium on Biomedical Imaging (ISBI)},
}
le Roux, V., Davel, M. H., & Bosman, J. (2026). A deep learning framework for expert-guided sailplane performance optimisation. In (200685th ed., Vol. 31). Intelligent Systems with Applications . http://doi.org/10.1016/j.iswa.2026.200685

The aerodynamic optimisation of high-performance aircraft is traditionally hampered by computationally prohibitive simulations, limiting the scope and agility of design space exploration. To overcome this barrier, we introduce a unified, data-driven framework founded on deep neural networks that enables rapid, expert-guided optimisation. The framework replaces the conventional simulation pipeline with a suite of bespoke deep learning models: one for the controllable generation of airfoils from intuitive geometric parameters, and another for their instantaneous 2D performance prediction. Applied to the optimisation of a state-of-the-art 15-meter standard class sailplane, the framework achieved a mean lift-to-drag ratio increase of 1.26% over a competitive baseline, while rigorously adhering to all user-defined geometric constraints. The results demonstrate that combining expert-defined geometric constraints with data-driven design exploration can substantially reduce computational overhead while maintaining physical interpretability in preliminary aerodynamic design.

@inbook{612,
  author = {Vincent le Roux and Marelie Davel and Johan Bosman},
  title = {A deep learning framework for expert-guided sailplane performance optimisation},
  abstract = {The aerodynamic optimisation of high-performance aircraft is traditionally hampered by computationally prohibitive simulations, limiting the scope and agility of design space exploration. To overcome this barrier, we introduce a unified, data-driven framework founded on deep neural networks that enables rapid, expert-guided optimisation. The framework replaces the conventional simulation pipeline with a suite of bespoke deep learning models: one for the controllable generation of airfoils from intuitive geometric parameters, and another for their instantaneous 2D performance prediction. Applied to the optimisation of a state-of-the-art 15-meter standard class sailplane, the framework achieved a mean lift-to-drag ratio increase of 1.26% over a competitive baseline, while rigorously adhering to all user-defined geometric constraints. The results demonstrate that combining expert-defined geometric constraints with data-driven design exploration can substantially reduce computational overhead while maintaining physical interpretability in preliminary aerodynamic design.},
  year = {2026},
  volume = {31},
  edition = {200685},
  month = {09/2026},
  publisher = {Intelligent Systems with Applications},
  doi = {10.1016/j.iswa.2026.200685},
}
Smith, M. E., Bernard, S., Fearon, G., Krug, M., Molapo, R., Veitch, J., … Williams, L. (2026). Bridging the in situ data gap: a satellite- and model-based framework for operational monitoring of marine harmful algal blooms and temperature-related threats in emerging economies. In (1934010th ed., Vol. 13). Frontiers in Marine Science. http://doi.org/10.3389/fmars.2026.1934010

Emerging and developing economies often face constrained financial, technical, and institutional capacities to establish and maintain the ocean-based in situ observational networks that underpin marine decision support applications. This limitation heightens their exposure and sensitivity to the cumulative impacts of marine environmental hazards, including harmful algal blooms and marine heatwaves. In this perspective, we present an example of an operational, co-developed ocean monitoring framework that is explicitly structured around user requirements, local scientific expertise, established software engineering practices, and openly available satellite and numerical model data and algorithms. The framework is instantiated through the South African National Oceans and Coastal Information Management System (OCIMS) Fisheries and Aquaculture Decision Support Tool, which is employed as a representative implementation case study and potential reference architecture for application in other regional contexts. We demonstrate that satellite-derived and model-based products can markedly accelerate the establishment of operational marine environmental monitoring capabilities in settings where dense, long-term in situ observing systems are absent or sparse, thereby enhancing societal preparedness and adaptive capacity in relation to intensifying, climate-driven marine risks. Furthermore, we discuss how such frameworks can underpin resilient, equitable, and evidence-based blue economy development in the Global South.

@inbook{611,
  author = {Marié Smith and Stewart Bernard and Giles Fearon and Marjolaine Krug and Raymond Molapo and Jennifer Veitch and Lufuno Vhengani and Lauren Williams},
  title = {Bridging the in situ data gap: a satellite- and model-based framework for operational monitoring of marine harmful algal blooms and temperature-related threats in emerging economies},
  abstract = {Emerging and developing economies often face constrained financial, technical, and institutional capacities to establish and maintain the ocean-based in situ observational networks that underpin marine decision support applications. This limitation heightens their exposure and sensitivity to the cumulative impacts of marine environmental hazards, including harmful algal blooms and marine heatwaves. In this perspective, we present an example of an operational, co-developed ocean monitoring framework that is explicitly structured around user requirements, local scientific expertise, established software engineering practices, and openly available satellite and numerical model data and algorithms. The framework is instantiated through the South African National Oceans and Coastal Information Management System (OCIMS) Fisheries and Aquaculture Decision Support Tool, which is employed as a representative implementation case study and potential reference architecture for application in other regional contexts. We demonstrate that satellite-derived and model-based products can markedly accelerate the establishment of operational marine environmental monitoring capabilities in settings where dense, long-term in situ observing systems are absent or sparse, thereby enhancing societal preparedness and adaptive capacity in relation to intensifying, climate-driven marine risks. Furthermore, we discuss how such frameworks can underpin resilient, equitable, and evidence-based blue economy development in the Global South.},
  year = {2026},
  volume = {13},
  edition = {1934010},
  month = {09/2026},
  publisher = {Frontiers in Marine Science},
  doi = {10.3389/fmars.2026.1934010},
}
Ngorima, A., Helberg, A., & Davel, M. H. (2026). Convolutional DPA Refinement for Real-Time Channel Estimation in High-Mobility Vehicular Networks. In (Vol. 14). IEEE Access. http://doi.org/https://doi.org/10.1109/ACCESS.2026.3727327

Channel estimation in IEEE 802.11p vehicular networks must maintain reliable accuracy under severe Doppler conditions while meeting the receiver processing-time requirements of continuous frame reception. Although recurrent neural network (RNN)-based estimators can achieve competitive accuracy, their sequential hidden-state propagation leads to high per-frame latency that limits real-time applicability. To address this challenge, this paper proposes a two-part framework. First, a 1D residual dilated CNN (RDCNN) refines data pilot-aided (DPA) channel estimates across complete OFDM frames in a single parallel forward pass, avoiding the sequential per-symbol processing that dominates RNN-based latency. Second, a multi- channel mixed signal-to-noise ratio (SNR) training strategy exposes the estimator to diverse propagation environments and noise levels simultaneously, yielding a single model that generalises to unseen channel conditions without retraining or online adaptation. Simulation results across six Acosta-Marum vehicular channel models, including three reserved exclusively for testing, show that DPA-RDCNN with multi-channel mixed-SNR training surpasses RNN-based baselines in accuracy while reducing per-frame processing time by a factor of 16 to 32.

@inbook{610,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {Convolutional DPA Refinement for Real-Time Channel Estimation in High-Mobility Vehicular Networks},
  abstract = {Channel estimation in IEEE 802.11p vehicular networks must maintain reliable accuracy under severe Doppler conditions while meeting the receiver processing-time requirements of continuous frame reception. Although recurrent neural network (RNN)-based estimators can achieve competitive accuracy, their sequential hidden-state propagation leads to high per-frame latency that limits real-time applicability. To address this challenge, this paper proposes a two-part framework. First, a 1D residual dilated CNN (RDCNN) refines data pilot-aided (DPA) channel estimates across complete OFDM frames in a single parallel forward pass, avoiding the sequential per-symbol processing that dominates RNN-based latency. Second, a multi- channel mixed signal-to-noise ratio (SNR) training strategy exposes the estimator to diverse propagation environments and noise levels simultaneously, yielding a single model that generalises to unseen channel conditions without retraining or online adaptation. Simulation results across six Acosta-Marum vehicular channel models, including three reserved exclusively for testing, show that DPA-RDCNN with multi-channel mixed-SNR training surpasses RNN-based baselines in accuracy while reducing per-frame processing time by a factor of 16 to 32.},
  year = {2026},
  volume = {14},
  pages = {137309 - 137327},
  month = {25 August 2026},
  publisher = {IEEE Access},
  doi = {https://doi.org/10.1109/ACCESS.2026.3727327},
}

2025

Theunissen, M. W., Rabe, R., & Davel, M. (2025). KnowIt: Deep Time Series Modeling and Interpretation. ArXiv.

KnowIt (Knowledge discovery in time series data) is a flexible framework for building deep time series models and interpreting them. It is implemented as a Python toolkit, with source code and documentation available from this https URL. It imposes minimal assumptions about task specifications and decouples the definition of dataset, deep neural network architecture, and interpretability technique through well defined interfaces. This ensures the ease of importing new datasets, custom architectures, and the definition of different interpretability paradigms while maintaining on-the-fly modeling and interpretation of different aspects of a user's own time series data. KnowIt aims to provide an environment where users can perform knowledge discovery on their own complex time series data through building powerful deep learning models and explaining their behavior. With ongoing development, collaboration and application our goal is to make this a platform to progress this underexplored field and produce a trusted tool for deep time series modeling.

@article{531,
  author = {Marthinus Theunissen and Randle Rabe and Marelie Davel},
  title = {KnowIt: Deep Time Series Modeling and Interpretation},
  abstract = {KnowIt (Knowledge discovery in time series data) is a flexible framework for building deep time series models and interpreting them. It is implemented as a Python toolkit, with source code and documentation available from this https URL. It imposes minimal assumptions about task specifications and decouples the definition of dataset, deep neural network architecture, and interpretability technique through well defined interfaces. This ensures the ease of importing new datasets, custom architectures, and the definition of different interpretability paradigms while maintaining on-the-fly modeling and interpretation of different aspects of a user's own time series data. KnowIt aims to provide an environment where users can perform knowledge discovery on their own complex time series data through building powerful deep learning models and explaining their behavior. With ongoing development, collaboration and application our goal is to make this a platform to progress this underexplored field and produce a trusted tool for deep time series modeling.},
  year = {2025},
  journal = {arXiv},
}
Van der Spoel, R., & Rabe, R. (2025). Investigating the relationship between diversity and generalization in deep neural networks. Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), 307.

In ensembles, improved generalization is frequently attributed to \emph{diversity} among members of the ensemble. By viewing a single neural network as an \emph{implicit ensemble}, we perform an exploratory investigation that applies well-known ensemble diversity measures to a neural network in order to study the relationship between diversity and generalization in the over-parameterized regime. Our results show that i) deeper layers of the network generally have higher levels of diversity—particularly for MLPs—and ii) layer-wise accuracy positively correlates with diversity. Additionally, we study the effects of well-known regularizers such as Dropout, DropConnect and batch size, on diversity and generalization. We generally find that increasing the strength of the regularizer increases the diversity in the neural network and this increase in diversity is positively correlated with model accuracy. We show that these results hold for several benchmark datasets (such as Fashion-MNIST and CIFAR-10) and architectures (MLPs and CNNs). Our findings suggest new avenues of research into the generalization ability of deep neural networks.

@article{530,
  author = {Ruan Van der Spoel and Randle Rabe},
  title = {Investigating the relationship between diversity and generalization in deep neural networks},
  abstract = {In ensembles, improved generalization is frequently attributed to \emph{diversity} among members of the ensemble. By viewing a single neural network as an \emph{implicit ensemble}, we perform an exploratory investigation that applies well-known ensemble diversity measures to a neural network in order to study the relationship between diversity and generalization in the over-parameterized regime. Our results show that i) deeper layers of the network generally have higher levels of diversity—particularly for MLPs—and ii) layer-wise accuracy positively correlates with diversity. Additionally, we study the effects of well-known regularizers such as Dropout, DropConnect and batch size, on diversity and generalization. We generally find that increasing the strength of the regularizer increases the diversity in the neural network and this increase in diversity is positively correlated with model accuracy. We show that these results hold for several benchmark datasets (such as Fashion-MNIST and CIFAR-10) and architectures (MLPs and CNNs). Our findings suggest new avenues of research into the generalization ability of deep neural networks.},
  year = {2025},
  journal = {Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL)},
  volume = {307},
  pages = {375 - 387},
}
Freese, L., & Theunissen, M. W. (2025). Data Pruning: Redundant, Problematic, and Interdependent Samples. Artificial Intelligence Research. SACAIR 2025. Communications in Computer and Information Science, vol 2784. http://doi.org/https://doi.org/10.1007/978-3-032-11733-5_12

The performance of deep learning models is affected by not only data quantity but also data quality. Data pruning is a process by which practitioners can reduce the size of a dataset by only keeping the most important training data points, thereby achieving similar test set performance. We empirically investigate two popular data pruning methods under noisy and noiseless conditions and show that these methods fail in the presence of significant label noise. We highlight that the success of data pruning is distinctly affected by three factors: redundancy in the dataset, the presence of problematic samples, and interdependence between samples. We perform a detailed investigation on commonly used benchmark classification datasets and neural network architectures. We find that our observations are consistent across data distributions and training protocols.

@article{529,
  author = {Leon Freese and Marthinus Theunissen},
  title = {Data Pruning: Redundant, Problematic, and Interdependent Samples},
  abstract = {The performance of deep learning models is affected by not only data quantity but also data quality. Data pruning is a process by which practitioners can reduce the size of a dataset by only keeping the most important training data points, thereby achieving similar test set performance. We empirically investigate two popular data pruning methods under noisy and noiseless conditions and show that these methods fail in the presence of significant label noise. We highlight that the success of data pruning is distinctly affected by three factors: redundancy in the dataset, the presence of problematic samples, and interdependence between samples. We perform a detailed investigation on commonly used benchmark classification datasets and neural network architectures. We find that our observations are consistent across data distributions and training protocols.},
  year = {2025},
  journal = {Artificial Intelligence Research. SACAIR 2025. Communications in Computer and Information Science},
  volume = {vol 2784},
  month = {25 November 2025},
  doi = {https://doi.org/10.1007/978-3-032-11733-5_12},
}
Ngorima, A., Helberg, A., & Davel, M. (2025). Feature extraction for plant growth estimation. Artificial Intelligence Research. SACAIR 2025. Communications in Computer and Information Science, vol 2784. http://doi.org/https://doi.org/10.1007/978-3-032-11733-5_5

Precision agriculture requires the estimation of plant growth stages in real-time. When the plant growth stage is known, the wastage of resources in cultivation, such as nutrients and water, is reduced as only the required resources need to be supplied. Plants at different growth stages, however, have similar morphological features, which can make autonomous growth stage estimation difficult. This paper presents two feature extraction methods for growth stage estimation: one that uses a bank of Gabor filters and morphological operations, and the other that uses pre-trained convolutional neural networks (CNNs) and transfer learning. We test these methods on a publicly available plant growth stage dataset (“bccr-segset“) for two species, canola and radish, grown and captured under indoor conditions. The two proposed feature extraction methods are compared, using support vector machines and boosted trees as classifiers. We find that both methods are suitable for real-time applications, and that CNN features outperform the hand-crafted features, both with regard to speed and accuracy. The best system (VGG-19 features, classified with a radial basis function support vector machine) obtained an accuracy of 98.4% for both species, processing an image in 0.08 seconds.

@article{528,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {Feature extraction for plant growth estimation},
  abstract = {Precision agriculture requires the estimation of plant growth stages in real-time. When the plant growth stage is known, the wastage of resources in cultivation, such as nutrients and water, is reduced as only the required resources need to be supplied. Plants at different growth stages, however, have similar morphological features, which can make autonomous growth stage estimation difficult. This paper presents two feature extraction methods for growth stage estimation: one that uses a bank of Gabor filters and morphological operations, and the other that uses pre-trained convolutional neural networks (CNNs) and transfer learning. We test these methods on a publicly available plant growth stage dataset (“bccr-segset“) for two species, canola and radish, grown and captured under indoor conditions. The two proposed feature extraction methods are compared, using support vector machines and boosted trees as classifiers. We find that both methods are suitable for real-time applications, and that CNN features outperform the hand-crafted features, both with regard to speed and accuracy. The best system (VGG-19 features, classified with a radial basis function support vector machine) obtained an accuracy of 98.4% for both species, processing an image in 0.08 seconds.},
  year = {2025},
  journal = {Artificial Intelligence Research. SACAIR 2025. Communications in Computer and Information Science},
  volume = {vol 2784},
  month = {25 November 2025},
  doi = {https://doi.org/10.1007/978-3-032-11733-5_5},
}
Haupt, S., Sibolla, B., Molapo, R., Mdakane, L., & Fourie, N. (2025). Exploring the Use of Data in a Digital Twin for the Marine and Coastal Environment. International Jornal of Geo-Information, 14(4). http://doi.org/https://doi.org/10.3390/ijgi14040140

The ocean plays a vital role in our society and represents a constantly changing landscape that is not well understood and therefore needs continuous monitoring and research. Sustainable monitoring is essential to assess both the current and future state of our oceans. However, conventional monitoring faces significant challenges, including issues of accessibility, and spatial and temporal constraints. The development of digital twins of the ocean (DTO) offers an emerging technology that could revolutionise our understanding of marine and coastal environments. Current DTO have shown effectiveness in monitoring marine and coastal environments in the European context. However, there is a need for a DTO for the Southern African and Western Indian Ocean regions that addresses specific concerns that are relevant to these regions. Successful development of a DTO depends on the availability of high-quality data. Therefore, various data inputs are necessary to build an accurate digital twin. This paper explores the data that can be utilised in a DTO, detailing how different ocean variables are collected and integrated into the digital twin. As a first step towards the development of a DTO in these regions, the paper proposes a data management plan and its implementation in the development of DTO. The data management plan is based on the phases of data in a geospatial data life cycle. Challenges regarding the management of data in this DTO and possible solutions are presented in the conclusion.

@article{522,
  author = {Shelley Haupt and Bolelang Sibolla and Raymond Molapo and Lizwe Mdakane and Nicolene Fourie},
  title = {Exploring the Use of Data in a Digital Twin for the Marine and Coastal Environment},
  abstract = {The ocean plays a vital role in our society and represents a constantly changing landscape that is not well understood and therefore needs continuous monitoring and research. Sustainable monitoring is essential to assess both the current and future state of our oceans. However, conventional monitoring faces significant challenges, including issues of accessibility, and spatial and temporal constraints. The development of digital twins of the ocean (DTO) offers an emerging technology that could revolutionise our understanding of marine and coastal environments. Current DTO have shown effectiveness in monitoring marine and coastal environments in the European context. However, there is a need for a DTO for the Southern African and Western Indian Ocean regions that addresses specific concerns that are relevant to these regions. Successful development of a DTO depends on the availability of high-quality data. Therefore, various data inputs are necessary to build an accurate digital twin. This paper explores the data that can be utilised in a DTO, detailing how different ocean variables are collected and integrated into the digital twin. As a first step towards the development of a DTO in these regions, the paper proposes a data management plan and its implementation in the development of DTO. The data management plan is based on the phases of data in a geospatial data life cycle. Challenges regarding the management of data in this DTO and possible solutions are presented in the conclusion.},
  year = {2025},
  journal = {International Jornal of Geo-Information},
  volume = {14},
  month = {03/2025},
  issue = {4},
  doi = {https://doi.org/10.3390/ijgi14040140},
}
Ngorima, A., Helberg, A., & Davel, M. H. (2025). Simplified Temporal Convolutional-Based Channel Estimation for a WiFi Vehicular Communication Channel. IEEE 3rd Wireless Africa Conference (WAC). http://doi.org/10.1109/WAC63911.2025.10992609

@article{521,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {Simplified Temporal Convolutional-Based Channel Estimation for a WiFi Vehicular Communication Channel},
  abstract = {},
  year = {2025},
  journal = {IEEE 3rd Wireless Africa Conference (WAC)},
  pages = {1 - 5},
  month = {02/2025},
  publisher = {IEEE},
  address = {Pretoria, South Africa},
  isbn = {979-8-3315-1758-8},
  doi = {10.1109/WAC63911.2025.10992609},
}

2024

Brooks, W., Davel, M. H., & Mouton, C. (2024). Does Simple Trump Complex? Comparing Strategies for Adversarial Robustness in DNNs. Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science, vol 2326. http://doi.org/https://doi.org/10.1007/978-3-031-78255-8_15

@article{520,
  author = {William Brooks and Marelie Davel and Coenraad Mouton},
  title = {Does Simple Trump Complex? Comparing Strategies for Adversarial Robustness in DNNs},
  abstract = {},
  year = {2024},
  journal = {Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science},
  volume = {vol 2326},
  pages = {253 - 269},
  month = {12/2024},
  publisher = {Springer Nature Switzerland},
  address = {Cham},
  doi = {https://doi.org/10.1007/978-3-031-78255-8_15},
}
Potgieter, H., Mouton, C., & Davel, M. H. (2024). Impact of Batch Normalization on Convolutional Network Representations. Artificial Intelligence Research (SACAIR 2024), vol 2326. http://doi.org/https://doi.org/10.1007/978-3-031-78255-8_14

Batch normalization (BatchNorm) is a popular layer normalization technique used when training deep neural networks. It has been shown to enhance the training speed and accuracy of deep learning models. However, the mechanics by which BatchNorm achieves these benefits is an active area of research, and different perspectives have been proposed. In this paper, we investigate the effect of BatchNorm on the resulting hidden representations, that is, the vectors of activation values formed as samples are processed at each hidden layer. Specifically, we consider the sparsity of these representations, as well as their implicit clustering – the creation of groups of representations that are similar to some extent. We contrast image classification models trained with and without batch normalization and highlight consistent differences observed. These findings highlight that BatchNorm’s effect on representational sparsity is not a significant factor affecting generalization, while the representations of models trained with BatchNorm tend to show more advantageous clustering characteristics.

@article{518,
  author = {Harmen Potgieter and Coenraad Mouton and Marelie Davel},
  title = {Impact of Batch Normalization on Convolutional Network Representations},
  abstract = {Batch normalization (BatchNorm) is a popular layer normalization technique used when training deep neural networks. It has been shown to enhance the training speed and accuracy of deep learning models. However, the mechanics by which BatchNorm achieves these benefits is an active area of research, and different perspectives have been proposed. In this paper, we investigate the effect of BatchNorm on the resulting hidden representations, that is, the vectors of activation values formed as samples are processed at each hidden layer. Specifically, we consider the sparsity of these representations, as well as their implicit clustering – the creation of groups of representations that are similar to some extent. We contrast image classification models trained with and without batch normalization and highlight consistent differences observed. These findings highlight that BatchNorm’s effect on representational sparsity is not a significant factor affecting generalization, while the representations of models trained with BatchNorm tend to show more advantageous clustering characteristics.},
  year = {2024},
  journal = {Artificial Intelligence Research (SACAIR 2024)},
  volume = {vol 2326},
  pages = {235 - 252},
  month = {12/2024},
  publisher = {Springer Nature Switzerland},
  address = {Cham},
  doi = {https://doi.org/10.1007/978-3-031-78255-8_14},
}
Ramalepe, S., Modipa, T., & Davel, M. H. (2024). Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset. Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science, vol 2326. http://doi.org/https://doi.org/10.1007/978-3-031-78255-8_19

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.

@article{517,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset},
  abstract = {Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.},
  year = {2024},
  journal = {Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science},
  volume = {vol 2326},
  pages = {319-333},
  month = {12/2024},
  publisher = {Springer Nature Switzerland},
  address = {Cham},
  doi = {https://doi.org/10.1007/978-3-031-78255-8_19},
}
Ngorima, A., Helberg, A., & Davel, M. H. (2024). Neural Network-Based Vehicular Channel Estimation Performance: Effect of Noise in the Training Set. Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science, vol 2326. http://doi.org/https://doi.org/10.1007/978-3-031-78255-8_12

Vehicular communication systems face significant challenges due to high mobility and rapidly changing environments, which affect the channel over which the signals travel. To address these challenges, neural network (NN)-based channel estimation methods have been suggested. These methods are primarily trained on high signal-to-noise ratio (SNR) with the assumption that training a NN in less noisy conditions can result in good generalisation. This study examines the effectiveness of training NN-based channel estimators on mixed SNR datasets compared to training solely on high SNR datasets, as seen in several related works. Estimators evaluated in this work include an architecture that uses convolutional layers and self-attention mechanisms; a method that employs temporal convolutional networks and data pilot-aided estimation; two methods that combine classical methods with multilayer perceptrons; and the current state-of-the-art model that combines Long-Short-Term Memory networks with data pilot-aided and temporal averaging methods as post processing. Our results indicate that using only high SNR data for training is not always optimal, and the SNR range in the training dataset should be treated as a hyperparameter that can be adjusted for better performance. This is illustrated by the better performance of some models in low SNR conditions when trained on the mixed SNR dataset, as opposed to when trained exclusively on high SNR data.

@article{516,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {Neural Network-Based Vehicular Channel Estimation Performance: Effect of Noise in the Training Set},
  abstract = {Vehicular communication systems face significant challenges due to high mobility and rapidly changing environments, which affect the channel over which the signals travel. To address these challenges, neural network (NN)-based channel estimation methods have been suggested. These methods are primarily trained on high signal-to-noise ratio (SNR) with the assumption that training a NN in less noisy conditions can result in good generalisation. This study examines the effectiveness of training NN-based channel estimators on mixed SNR datasets compared to training solely on high SNR datasets, as seen in several related works. Estimators evaluated in this work include an architecture that uses convolutional layers and self-attention mechanisms; a method that employs temporal convolutional networks and data pilot-aided estimation; two methods that combine classical methods with multilayer perceptrons; and the current state-of-the-art model that combines Long-Short-Term Memory networks with data pilot-aided and temporal averaging methods as post processing. Our results indicate that using only high SNR data for training is not always optimal, and the SNR range in the training dataset should be treated as a hyperparameter that can be adjusted for better performance. This is illustrated by the better performance of some models in low SNR conditions when trained on the mixed SNR dataset, as opposed to when trained exclusively on high SNR data.},
  year = {2024},
  journal = {Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science},
  volume = {vol 2326},
  pages = {192 - 206},
  month = {12/2024},
  publisher = {Springer Nature Switzerland},
  address = {Cham},
  isbn = {978-3-031-78255-8},
  doi = {https://doi.org/10.1007/978-3-031-78255-8_12},
}
Ngorima, A., Helberg, A., & Davel, M. H. (2024). A Data Pilot-Aided Temporal Convolutional Network for Channel Estimation in IEEE 802.11p Vehicle-to-Vehicle Communications. Southern Africa Telecommunication Networks and Applications Conference (SATNAC).

In modern communication systems, having an accurate channel estimator is crucial. However, when there is mobility, it becomes difficult to estimate the channel and the pilot signals, which are used for channel estimation, become insufficient. In this paper, we introduce the use of Temporal
Convolutional Networks (TCNs) with data pilot-aided (DPA) channel estimation and temporal averaging (TA) to estimate vehicle-to-vehicle same direction with Wall (VTV-SDWW) channels. The TCN-DPA-TA estimator showed an improvement in Bit Error Rate (BER) performance of up to 1 order of magnitude. Furthermore, the BER performance of the TCN-DPA without TA also improved by up to 0.7 magnitude compared to the best classical estimator.

@article{515,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {A Data Pilot-Aided Temporal Convolutional Network for Channel Estimation in IEEE 802.11p Vehicle-to-Vehicle Communications},
  abstract = {In modern communication systems, having an accurate channel estimator is crucial. However, when there is mobility, it becomes difficult to estimate the channel and the pilot signals, which are used for channel estimation, become insufficient. In this paper, we introduce the use of TemporalConvolutional Networks (TCNs) with data pilot-aided (DPA) channel estimation and temporal averaging (TA) to estimate vehicle-to-vehicle same direction with Wall (VTV-SDWW) channels. The TCN-DPA-TA estimator showed an improvement in Bit Error Rate (BER) performance of up to 1 order of magnitude. Furthermore, the BER performance of the TCN-DPA without TA also improved by up to 0.7 magnitude compared to the best classical estimator.},
  year = {2024},
  journal = {Southern Africa Telecommunication Networks and Applications Conference (SATNAC)},
  pages = {356–361},
}
  • DSI
  • Covid-19