Hi MalariaGen team (and @jonbrenas),
I have been auditing the gsoc-mosquito-taxon-classifier repository and the current state of the taxonomic classification pipeline. I see the foundation for feature extraction is laid out, but the predictive engine and the orchestration for large-scale processing are pending.
I am currently architecting a scalable machine learning pipeline as part of an advanced university research project, focusing specifically on the Ag1000G dataset. My objective is to implement a Random Forest classifier optimized for out-of-core computation using Dask and Zarr. This architecture will allow parallel processing of massive genomic arrays efficiently, minimizing RAM bottlenecks in environments like Google Colab.
I will be developing and iterating on this infrastructure over the coming months. Since my goal is to eventually contribute this predictive engine back to malariagen-data-python, I wanted to open this thread to ensure this technical approach aligns with your current roadmap for the taxonomy classifier.
Looking forward to your thoughts.
Best regards, Brayan
Hi MalariaGen team (and @jonbrenas),
I have been auditing the
gsoc-mosquito-taxon-classifierrepository and the current state of the taxonomic classification pipeline. I see the foundation for feature extraction is laid out, but the predictive engine and the orchestration for large-scale processing are pending.I am currently architecting a scalable machine learning pipeline as part of an advanced university research project, focusing specifically on the Ag1000G dataset. My objective is to implement a Random Forest classifier optimized for out-of-core computation using Dask and Zarr. This architecture will allow parallel processing of massive genomic arrays efficiently, minimizing RAM bottlenecks in environments like Google Colab.
I will be developing and iterating on this infrastructure over the coming months. Since my goal is to eventually contribute this predictive engine back to
malariagen-data-python, I wanted to open this thread to ensure this technical approach aligns with your current roadmap for the taxonomy classifier.Looking forward to your thoughts.
Best regards, Brayan