A high-throughput and memory-efficient inference and serving engine for LLMs

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

Community maintained hardware plugin for vLLM on Intel Gaudi

codes documenting my gpu learning journey

SOTA Weight-only Quantization Algorithm for LLMs

Samples for Intel® oneAPI Toolkits

SGLang is a fast serving framework for large language models and vision language models.

🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"

A safetensors extension to efficiently store sparse quantized tensors on disk

Development repository for the Triton language and compiler

A framework for few-shot evaluation of language models.

The torchao repository contains api's and workflows for quantization and pruning gpu models.

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.

PyTorch emulation library for Microscaling (MX)-compatible data formats

FlashInfer: Kernel Library for LLM Serving

Easy and lightning fast training of 🤗 Transformers on Habana Gaudi processor (HPU)

Tensors and Dynamic neural networks in Python with strong GPU acceleration

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform