Randle Rabe

Student Type: PhD

Publications

2024

Mouton, C. ., Rabe, R. ., Haasbroek, D. G., Theunissen, M. W., Potgieter, H. ., & Davel, M. H. (2024). Is network fragmentation a useful complexity measure?. NeurIPS 2024 Workshop SciForDL.

Abstract BibTex Entry BibTeX Download PDF

27_Is_network_fragmentation_a_.pdf

It has been observed that the input space of deep neural network classifiers can exhibit ‘fragmentation’, where the model function rapidly changes class as the input space is traversed. The severity of this fragmentation tends to follow the double descent curve, achieving a maximum at the interpolation regime. We study this phenomenon in the context of image classification and ask whether fragmentation could be predictive of generalization performance. Using a fragmentation-based complexity measure, we show this to be possible by achieving good performance on the PGDL (Predicting Generalization in Deep Learning) benchmark. In addition, we report on new observations related to fragmentation, namely (i) fragmentation is not limited to the input space but occurs in the hidden representations as well, (ii) fragmentation follows the trends in the validation error throughout training, and (iii) fragmentation is not a direct result of increased weight norms. Together, this indicates that fragmentation is a phenomenon worth investigating further when studying the generalization ability of deep neural networks.

@misc{514,
  author = {Coenraad Mouton and Randle Rabe and Daniël Haasbroek and Marthinus Theunissen and Harmen Potgieter and Marelie Davel},
  title = {Is network fragmentation a useful complexity measure?},
  abstract = {It has been observed that the input space of deep neural network classifiers can exhibit ‘fragmentation’, where the model function rapidly changes class as the input space is traversed. The severity of this fragmentation tends to follow the double descent curve, achieving a maximum at the interpolation regime. We study this phenomenon in the context of image classification and ask whether fragmentation could be predictive of generalization performance. Using a fragmentation-based complexity measure, we show this to be possible by achieving good performance on the PGDL (Predicting Generalization in Deep Learning) benchmark. In addition, we report on new observations related to fragmentation, namely (i) fragmentation is not limited to the input space but occurs in the hidden representations as well, (ii) fragmentation follows the trends in the validation error throughout training, and (iii) fragmentation is not a direct result of increased weight norms. Together, this indicates that fragmentation is a phenomenon worth investigating further when studying the generalization ability of deep neural networks.},
  year = {2024},
  journal = {NeurIPS 2024 Workshop SciForDL},
  month = {12/2024},
}

2023

Davel, M. H., Lotz, S. ., Theunissen, M. W., De Villiers, A. ., Grant, C. ., Rabe, R. ., & Conacher, C. . (2023). Knowledge Discovery in Time Series Data. In Deep Learning Indaba 2023.

Abstract BibTex Entry BibTeX Download PDF

DLI_poster_2023.pdf

• Complex time series data often encountered in scientific and engineering domains. • Deep learning (DL) is particularly successful here: – large data sets, multivariate input and/or ouput, – highly complex sequences of interactions. • Model interpretability: – Ability to understand a model’s decisions in a given context [1]. – Techniques typically not originally developed for time series data. – Time series interpretations themselves become uninterpretable. • Knowledge Discovery: – DL has potential to reveal interesting patterns in large data sets. – Potential to produce novel insights about the task itself [2, 3]. • ‘know-it’: Collaborative project that studies knowledge discovery in time series data.

@{507,
  author = {Marelie Davel and Stefan Lotz and Marthinus Theunissen and Almaro De Villiers and Chara Grant and Randle Rabe and Cleo Conacher},
  title = {Knowledge Discovery in Time Series Data},
  abstract = {• Complex time series data often encountered in scientific and engineering domains.
• Deep learning (DL) is particularly successful here:
– large data sets, multivariate input and/or ouput,
– highly complex sequences of interactions.
• Model interpretability:
– Ability to understand a model’s decisions in a given context [1].
– Techniques typically not originally developed for time series data.
– Time series interpretations themselves become uninterpretable.
• Knowledge Discovery:
– DL has potential to reveal interesting patterns in large data sets.
– Potential to produce novel insights about the task itself [2, 3].
• ‘know-it’: Collaborative project that studies knowledge discovery in
time series data.},
  year = {2023},
  journal = {Deep Learning Indaba 2023},
  month = {September 2023},
}