DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
-
Updated
Jun 12, 2024 - Python
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
PyTorch library for cost-effective, fast and easy serving of MoE models.
Efficient global optimization toolbox in Rust: bayesian optimization, mixture of gaussian processes, sampling methods
pytorch implementation of grok
Surrogate Modeling Toolbox
Tutel MoE: An Optimized Mixture-of-Experts Implementation
[ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"
Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
This is the official repository of the papers "Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers" and "Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters".
[Preprint] Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
Repository for "See More Details: Efficient Image Super-Resolution by Experts Mining", ICML 2024
MoE Decoder Transformer implementation with MLX
[arXiv'24] Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
The idea to create the perfect LLM currently possible came to my mind because I was watching a YouTube on GaLore, the "sequel" to LoRa, and I realized how fucking groundbreaking that tech is. I was daydreaming about pretraining my own model, this (probably impossible to implement) concept is a refined version of that model.
[SIGIR'24] The official implementation code of MOELoRA.
[Paper][Preprint 2024] Mixture of Modality Knowledge Experts for Robust Multi-modal Knowledge Graph Completion
an LLM toolkit
Mistral and Mixtral (MoE) from scratch
Add a description, image, and links to the mixture-of-experts topic page so that developers can more easily learn about it.
To associate your repository with the mixture-of-experts topic, visit your repo's landing page and select "manage topics."