Simon Ramalepe

Student Type: PhD

Publications

2024

Ramalepe, S. ., Modipa, T. ., & Davel, M. H. (2024). Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset. Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science, vol 2326. http://doi.org/https://doi.org/10.1007/978-3-031-78255-8_19

Abstract BibTex Entry BibTeX Download PDF

2501.15281v1.pdf

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.

@article{517,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset},
  abstract = {Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.},
  year = {2024},
  journal = {Artificial Intelligence Research. SACAIR 2024. Communications in Computer and Information Science},
  volume = {vol 2326},
  pages = {319-333},
  month = {12/2024},
  publisher = {Springer Nature Switzerland},
  address = {Cham},
  doi = {https://doi.org/10.1007/978-3-031-78255-8_19},
}

2023

Ramalepe, S. ., Modipa, T. ., & Davel, M. H. (2023). The Analysis of the Sepedi-English Code-switched Radio News Corpus. Journal of the Digital Humanities Association of Southern Africa, 4(Vol. 4 No. 01 (2022): Proceedings of the 3rd workshop on Resources for African Indigenous Languages (RAIL). http://doi.org/https://doi.org/10.55492/dhasa.v4i01.4444

Abstract BibTex Entry BibTeX Download PDF

08_Radio+news+data+analysis_1.pdf

Code-switching is a phenomenon that occurs mostly in multilingual countries where multilingual speakers often switch between languages in their conversations. The unavailability of large scale code-switched corpora hampers the development and training of language models for the generation of code-switched text. In this study, we explore the initial phase of collecting and creating Sepedi-English code-switched corpus for generating synthetic news. Radio news and the frequency of code-switching on read news were considered and analysed. We developed and trained a Transformer-based language model using the collected code-switched dataset. We observed that the frequency of code-switched data in the dataset was very low at 1.1%. We complemented our dataset with the news headlines dataset to create a new dataset. Although the frequency was still low, the model obtained the optimal loss rate of 2,361 with an accuracy of 66%.

@article{502,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {The Analysis of the Sepedi-English Code-switched Radio News Corpus},
  abstract = {Code-switching is a phenomenon that occurs mostly in multilingual countries where multilingual speakers often switch between languages in
their conversations. The unavailability of large scale code-switched corpora hampers the development and training of language models for the generation of code-switched text. In this study, we explore the initial phase of collecting and creating Sepedi-English code-switched corpus for generating synthetic news. Radio news and the frequency of code-switching on read news were considered and analysed. We developed and trained a Transformer-based language model using the collected code-switched dataset. We observed that the frequency of code-switched data in the dataset was very low at 1.1%. We complemented our dataset with the news headlines dataset to create a new dataset.
Although the frequency was still low, the model obtained the optimal loss rate of 2,361 with an accuracy of 66%.},
  year = {2023},
  journal = {Journal of the Digital Humanities Association of Southern Africa},
  volume = {4},
  edition = {1},
  month = {2023-01-25},
  issue = {Vol. 4 No. 01 (2022): Proceedings of the 3rd workshop on Resources for African Indigenous Languages (RAIL)},
  doi = {https://doi.org/10.55492/dhasa.v4i01.4444},
}

Ramalepe, S. ., Modipa, T. ., & Davel, M. H. (2023). Transformer-based text generation for code-switched Sepedi-English news. In Southern African Conference for Artificial Intelligence Research (SACAIR).

Abstract BibTex Entry BibTeX Download PDF

36-sacair23.pdf

Code-switched data is rarely available in written form and this makes the development of large datasets required to train codeswitched language models difficult. Currently, available Sepedi-English code-switched corpora are not large enough to train a Transformer-based model for this language pair. In prior work, larger synthetic datasets have been constructed using a combination of a monolingual and a parallel corpus to approximate authentic code-switched text. In this study, we develop and analyse a new Sepedi-English news dataset (SepEnews). We collect and curate data from local radio news bulletins and use this to augment two existing sources collected from Sepedi newspapers and news headlines, respectively. We then develop and train a Transformer-based model for generating historic code-switched news, and demonstrate and analyse the system’s performance.

@{501,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {Transformer-based text generation for code-switched Sepedi-English news},
  abstract = {Code-switched data is rarely available in written form and this makes the development of large datasets required to train codeswitched language models difficult. Currently, available Sepedi-English code-switched corpora are not large enough to train a Transformer-based
model for this language pair. In prior work, larger synthetic datasets have been constructed using a combination of a monolingual and a parallel
corpus to approximate authentic code-switched text. In this study, we develop and analyse a new Sepedi-English news dataset (SepEnews). We collect and curate data from local radio news bulletins and use this to augment two existing sources collected from Sepedi newspapers and news headlines, respectively. We then develop and train a Transformer-based model for generating historic code-switched news, and demonstrate and analyse the system’s performance.},
  year = {2023},
  journal = {Southern African Conference for Artificial Intelligence Research (SACAIR)},
  pages = {84 - 97},
  month = {December 2023},
}

2022

Ramalepe, S. ., Modipa, T. ., & Davel, M. H. (2022). The development of a Sepedi text generation model using transformers. In Southern Africa Telecommunication Networks and Applications Conference (SATNAC).

Abstract BibTex Entry BibTeX Download PDF

Ramalepe-2022-development-sepedi-text.pdf

Text generation is one of the important sub-tasks of natural language generation (NLG), and aims to produce humanly readable text given some input text. Deep learning approaches based on neural networks have been proposed to solve text generation tasks. Although these models can generate text, they do not necessarily capture long-term dependencies accurately, making it difficult to coherently generate longer sentences. Transformer-based models have shown significant improvement in text generation. However, these models are computationally expensive and data hungry. In this study, we develop a Sepedi text generation model using a Transformerbased approach and explore its performance. The developed model has one Transformer block with causal masking on the attention layers and two separate embedding layers. To train the model, we use the National Centre for Human Language Technology (NCHLT) Sepedi text corpus. Our experimental setup varied the model embedding size, batch size and the sequence length. The final model was able to reconstruct unseen test data with 75% accuracy: the highest accuracy achieved to date, using a Sepedi corpus.

@{511,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {The development of a Sepedi text generation model using transformers},
  abstract = {Text generation is one of the important sub-tasks of natural language generation (NLG), and aims to produce humanly readable text given some input text. Deep learning approaches based on neural networks have been proposed to solve text generation tasks. Although these models can generate text, they do not necessarily capture long-term dependencies accurately, making it difficult to coherently generate longer sentences. Transformer-based models have shown significant improvement in text generation. However, these models are computationally expensive and data hungry. In this study, we develop a Sepedi text generation model using a Transformerbased approach and explore its performance. The developed model has one Transformer block with causal masking on the attention layers and two separate embedding layers. To train the model, we use the National Centre for Human Language Technology (NCHLT) Sepedi text corpus. Our experimental setup varied the model embedding size, batch size and the sequence length. The final model was able to reconstruct unseen test data with 75% accuracy: the highest accuracy achieved to date, using a Sepedi corpus.},
  year = {2022},
  journal = {Southern Africa Telecommunication Networks and Applications Conference (SATNAC)},
  pages = {51 - 56},
  month = {August 2022},
}