llama.cppggml-org/llama.cpp
LLM inference in C/C++ on CPUs and GPUsPersonalLocal inferenceCLILibrary / SDK★ 130kMITcommit 22 hours ago86worth
Best open-source local LLM libraries & SDKs
Open-source runtimes for running LLMs on your own hardware.
10 projects
vLLMvllm-project/vllm
High-throughput LLM serving engineEnterpriseLocal inferenceDocker / self-hostLibrary / SDK★ 93kApache-2.0commit 21 hours ago86worth
Unslothunslothai/unsloth
Fast local fine-tuning and running of LLMsPersonalEnterpriseLocal inferenceWebLibrary / SDK★ 77kApache-2.0commit 22 hours ago81worth
SGLangsgl-project/sglang
Fast serving framework for LLMsEnterpriseLocal inferenceDocker / self-hostLibrary / SDK★ 37kApache-2.0commit 21 hours ago81worth
OpenVINOopenvinotoolkit/openvino
Optimize and deploy AI inferenceEnterpriseLocal inferenceLibrary / SDK★ 11kApache-2.0commit yesterday80worth
whisper.cppggml-org/whisper.cpp
Whisper speech recognition in C/C++PersonalEnterpriseLocal inferenceMedia & photosCLILibrary / SDK★ 54kMITcommit yesterday79worth
AirLLMlyogavin/airllm
Run 70B models on a 4GB GPUPersonalLocal inferenceLibrary / SDK★ 35kApache-2.0commit 3 days ago73worth
RunAnywhere SDKsRunanywhereAI/runanywhere-sdks
Run AI locally on devicesPersonalEnterpriseLocal inferenceMobileLibrary / SDK★ 10kcommit 2 weeks ago70worth
ncnnTencent/ncnn
Neural network inference for mobilePersonalEnterpriseLocal inferenceLibrary / SDK★ 24kcommit 2 days ago66worth
BitNetmicrosoft/BitNet
Inference framework for 1-bit LLMsPersonalEnterpriseLocal inferenceLibrary / SDK★ 40kMITcommit 2 months ago63worth