rafska/awesome-local-llm：本地运行 LLM 的平台、工具与资源精选列表

A curated list of awesome platforms, tools, practices and resources that helps run LLMs locally

Inference platforms

Back to Table of Contents

Inference engines

Back to Table of Contents

User Interfaces

Back to Table of Contents

Large Language Models

Explorers, Benchmarks, Leaderboards

Back to Table of Contents

Model providers

Qwen - powered by Alibaba Cloud
Mistral AI - a pioneering French artificial intelligence startup
Tencent - a profile of a Chinese multinational technology conglomerate and holding company
Unsloth AI - focusing on making AI more accessible to everyone (GGUFs etc.)
bartowski - providing GGUF versions of popular LLMs
Beijing Academy of Artificial Intelligence - a private non-profit organization engaged in AI research and development
Open Thoughts - a team of researchers and engineers curating the best open reasoning datasets

Back to Table of Contents

Specific models

General purpose

DeepSeek-V4 - a collection of the DeepSeek V4 LLMs
Qwen3.8 - a collection of the latest generation Qwen LLMs
NVIDIA Nemotron v3 - a family of open models from NVIDIA with open weights, training data and recipes, delivering leading efficiency and accuracy for building specialized AI agents
Gemma 4 - a family of open models built by Google DeepMind, that are multimodal, handling text and image input (with audio supported on small models) and generating text output
Mistral Medium 3.5 - The first flaship models from Mistral AI handling instruction-following, reasoning, and coding in a single set of opened-weights
gpt-oss - a collection of open-weight models from OpenAI, designed for powerful reasoning, agentic tasks, and versatile developer use cases
gpt-oss-puzzle-88B - a deployment-optimized large language model developed by NVIDIA, derived from OpenAI's gpt-oss-120b
Hunyuan - a collection of Tencent's open-source efficient LLMs designed for versatile deployment across diverse computational environments
Phi-4 - a family of small language, multi-modal and reasoning models from Microsoft
OpenReasoning-Nemotron - a collection of models from NVIDIA, trained on 5M reasoning traces for math, code and science
Kimi K2.5 - a collection of open-source, native multimodal agentic models from Moonshot AI that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration
GLM-5.3 - a Z.ai's flagship model for long-horizon tasks
Ling-3.0-flash - a native hybrid reasoning model from inclusionAI, operating with 124B total and 5.1B active parameters
Granite 4.1 - efficient language models from IBM for multilingual generation, coding, RAG, and AI assistant workflows
Ornith-1.5 - a collection of open-source models for agentic tasks and coding
EXAONE-4.5 - LG's First Open-Weight Vision-Language Model for Industrial Intelligence
Step-3.7-Flash - Mixture-of-Experts vision-language model for developers who need to scale agentic workflows that combine perception, search, and reasoning
Occamy-1.0 - a compact agentic model purpose-built for real-world co-work
MiniCPM5 - a collection of SOTA on-device LLMs, small yet powerful

Back to Table of Contents

Coding

Back to Table of Contents

Multimodal

Qwen3-Omni - a collection of the natively end-to-end multilingual omni-modal foundation models from Qwen
GLM-4.6V - a collection of open source multimodal models with native tool use from Zhipu AI

Back to Table of Contents

Image

Qwen-Image-2.1 - a unified text-to-image generation and image editing model in the Qwen family
Qwen3-VL - a collection of the most powerful vision-language models in the Qwen series to date
GLM-Image - an image generation model
Ming-Image-0.1-Design - a 6B text-to-image model for UI, infographics, posters, and other text-rich visual designs
Granite Vision - multimodal models from IBM built for visual document analysis and image understanding
HunyuanImage - a collection of image generation models from Tencent
HunyuanVideo - a collection of video generation models from Tencent
Youtu-Parsing-Omni - a model from Tencent that parses documents, images, charts, geometry, audio and video into structured JSON
Vidi - a collection of models for multimodal video understanding and creation
FastVLM - a collection of VLMs with efficient vision encoding from Apple
MiniCPM-o & MiniCPM-V - multimodal models with leading performance
LFM2.5-VL - a collection of vision-language models, designed for on-device deployment

Back to Table of Contents

Audio

whisper-large-v3 - a state-of-the-art model for automatic speech recognition (ASR) and speech translation from OpenAI
Nemotron Speech - a collection of open, state-of-the-art, production‑ready enterprise speech models from NVIDIA for ASR, TTS, Speaker Diarization and S2SOpenAI
NVIDIA NemotronLabs VoiceChat 11B - a 11B end-to-end, real-time speech full duplex (FD) model from NVIDIA for conversational AI that jointly performs streaming speech understanding and speech generation
Qwen3-ASR - a collection of models that support language identification and ASR for 52 languages and dialects
Qwen3-TTS - a collection of TTS models that cover 10 major languages as well as multiple dialectal voice profiles to meet global application needs
Granite Speech - a collection of compact and efficient speech-language models from IBM, specifically designed for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST)
Voxtral-Small-24B-2507 - an enhancement of Mistral Small 3, incorporating state-of-the-art audio input capabilities while retaining best-in-class text performance
Voxtral-Mini-4B-Realtime-2602 - a multilingual, realtime speech-transcription model and among the first open-source solutions to achieve accuracy comparable to offline systems with a delay of <500ms
Voxtral-4B-TTS-2603 - frontier, open-weights text-to-speech model that’s fast, instantly adaptable, and produces lifelike speech for voice agents
chatterbox - first production-grade open-source TTS model
VibeVoice - a collection of frontier text-to-speech models from Microsoft
Kitten TTS - a collection of open-source realistic text-to-speech models designed for lightweight deployment and high-quality voice synthesis
Streaming Sortformer Diarizer 4spk v2.1 - a streaming version of a novel end-to-end neural model for speaker diarization from NVIDIA

Back to Table of Contents

Retrieval-Augmented Generation

Nemotron RAG - a set of tools to build retrieval-augmented generation (RAG) systems, improve search and ranking accuracy, and extract structured data from complex docs
EmbeddingGemma 2 - an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space
Qwen3-Embedding - a collection of the latest proprietary Qwen models, specifically designed for text embedding and ranking tasks
Qwen3-VL-Embedding - an addition to the Qwen embedding models, specifically designed for multimodal information retrieval and cross-modal understanding
Qwen3-Reranker - a collection of the latest proprietary Qwen models, engineered to refine embedding results
Qwen3-VL-Reranker - an addition to the Qwen embedding models, specifically designed for multimodal information retrieval and cross-modal understanding

Back to Table of Contents

Safeguards

Back to Table of Contents

Miscellaneous

Decision 3.0 - a collection of Open multimodal foundation decision models
GLiNER2.5 - a collection of models for schema-driven information extraction and classification
clef - a multimodal model from Cloudflare that turns a state and a schema of typed questions into decisions
d1-3B & d1-omni-600M - open source decision models
Intern-S2 - a collection of multimodal foundation models for scientific intelligence and long-horizon agents
Holo4 - a collection of Visual Language Models for computer/mobile use, tool calls, and code
Marco-MoE - a suit of multilingual MoE models with highly-sparse architectures
OUI-1 - a diffusion model built for generative UI
Nemotron-Orchestrator-8B - a state-of-the-art 8B orchestration model designed to solve complex, multi-turn agentic tasks by coordinating a diverse set of expert models and tools
Arch-Router-1.5B - the fastest LLM router model that aligns to subjective usage preferences
Waypoint - a collection of real-time interactive video world models
Hunyuan3D - a collection of everything related (models, datasets etc.) to 3D assets generation from Tencent
Hunyuan-GameCraft-1.0 - a novel framework for high-dynamic interactive video generation in game environments
void-model - a model from Netflix that removes objects from videos along with all interactions they induce on the scene — not just secondary effects like shadows and reflections, but physical interactions like objects falling when a person is removed

Back to Table of Contents

Tools

Models

Back to Table of Contents

Agent Frameworks

AutoGPT - a powerful platform that allows you to create, deploy, and manage continuous AI agents that automate complex workflows
langflow - a powerful tool for building and deploying AI-powered agents and workflows
langchain - build context-aware reasoning applications
pi - AI agent toolkit: coding agent CLI, unified LLM API, TUI & web UI libraries, Slack bot, vLLM pods
anything-llm - the all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more
autogen - a programming framework for agentic AI
crewAI - a framework for orchestrating role-playing, autonomous AI agents
Flowise - build AI agents, visually
llama_index - the leading framework for building LLM-powered agents over your data
agno - a full-stack framework for building Multi-Agent Systems with memory, knowledge and reasoning
sim - open-source platform to build and deploy AI agent workflows
openai-agents-python - a lightweight, powerful framework for multi-agent workflows
t3code - an agent harness control surface that enables control of the agents on your machine with a best-in-class mobile app
NemoClaw - run OpenClaw more securely inside NVIDIA OpenShell with managed inference
pydantic-ai - a Python agent framework designed to help you quickly, confidently, and painlessly build production grade applications and workflows with Generative AI
SuperAGI - an open-source framework to build, manage and run useful Autonomous AI Agents
camel - the first and the best multi-agent framework
agent-framework - a framework for building, orchestrating and deploying AI agents and multi-agent workflows with support for Python and .NET
superset - an agentic IDE to orchestrate 100+ coding agents in parallel
txtai - all-in-one open-source AI framework for semantic search, LLM orchestration and language model workflows
archgw - a high-performance proxy server that handles the low-level work in building agents: like applying guardrails, routing prompts to the right agent, and unifying access to LLMs, etc.
genkit - open-source framework for building AI-powered apps in JavaScript, Go, and Python, built and used in production by Google
emdash - a desktop app for running AI coding agents in parallel
ClaraVerse - privacy-first, fully local AI workspace with Ollama LLM chat, tool calling, agent builder, Stable Diffusion, and embedded n8n-style automation
NeMo-Agent-Toolkit - an open-source library for efficiently connecting and optimizing teams of AI agents
ragbits - building blocks for rapid development of GenAI applications

Back to Table of Contents

Model Context Protocol

Back to Table of Contents

Retrieval-Augmented Generation

Back to Table of Contents

Coding Agents

opencode - a AI coding agent built for the terminal
zed - a next-generation code editor designed for high-performance collaboration with humans and AI
OpenHands - a platform for software development agents powered by AI
cline - autonomous coding agent right in your IDE, capable of creating/editing files, executing commands, using the browser, and more with your permission every step of the way
goose - an open-source, extensible AI agent that goes beyond code suggestions
aider - AI pair programming in your terminal
continue - create, share, and use custom AI code assistants with our open-source IDE extensions and hub of models, rules, prompts, docs, and other building blocks
tabby - an open-source GitHub Copilot alternative, set up your own LLM-powered code completion server
void - an open-source Cursor alternative, use AI agents on your codebase, checkpoint and visualize changes, and bring any model or host locally
crush - the glamourous AI coding agent for your favourite terminal
kilocode - open source AI coding assistant for planning, building, and fixing code
Roo-Code - a whole dev team of AI agents in your code editor
humanlayer - the best way to get AI coding agents to solve hard problems in complex codebases
openchamber - Agentic Development Environment based on OpenCode AI agent
99 - neovim AI agent done right
ProxyAI - the leading open-source AI copilot for JetBrains

Back to Table of Contents

Testing, Evaluation and Observability

langfuse - an open-source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more
opik - debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards
openllmetry - an open-source observability for your LLM application, based on OpenTelemetry
giskard - an open-source evaluation & testing for AI & LLM systems
agenta - an open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place
Evaluator - open-source library for scalable, reproducible evaluation of AI models and benchmarks

Back to Table of Contents

Research

Back to Table of Contents

Training and Fine-tuning

Back to Table of Contents

Security and Sandboxing

Back to Table of Contents

Miscellaneous

context7 - up-to-date code documentation for LLMs and AI code editors
deepwiki-open - open source DeepWiki: AI-powered wiki generator for GitHub/Gitlab/Bitbucket repositories
cai - Cybersecurity AI (CAI), the framework for AI Security
speakr - a personal, self-hosted web application designed for transcribing audio recordings
presenton - an open-source AI presentation generator and API
OmniGen2 - exploration to advanced multimodal generation
4o-ghibli-at-home - a powerful, self-hosted AI photo stylizer built for performance and privacy
Observer - local open-source micro-agents that observe, log and react, all while keeping your data private and secure
mobile-use - a powerful, open-source AI agent that controls your Android or IOS device using natural language
gabber - build AI applications that can see, hear, and speak using your screens, microphones, and cameras as inputs
promptcat - a zero-dependency prompt manager/catalog/library in a single HTML file

Back to Table of Contents

Hardware

Back to Table of Contents

Tutorials

Models

Back to Table of Contents

Prompt Engineering

Back to Table of Contents

Context Engineering

Back to Table of Contents

Inference

vLLM Production Stack - vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

Back to Table of Contents

Agents

superpowers - an agentic skills framework & software development methodology that works
GenAI Agents - tutorials and implementations for various Generative AI Agent techniques
500+ AI Agent Projects - a curated collection of AI agent use cases across various industries
12-Factor Agents - principles for building reliable LLM applications
Agents towards production - end-to-end, code-first tutorials covering every layer of production-grade GenAI agents, guiding you from spark to scale with proven patterns and reusable blueprints for real-world launches
agents.md - a simple, open format for guiding coding agents
Agent Skills - a simple, open format for giving agents new capabilities and expertise
skills - Hugging Face Skills are definitions for AI/ML tasks like dataset creation, model training and evaluation
LLM Agents & Ecosystem Handbook - one-stop handbook for building, deploying, and understanding LLM agents with 60+ skeletons, tutorials, ecosystem guides, and evaluation tools
601 real-world gen AI use cases - 601 real-world gen AI use cases from the world's leading organizations by Google
A practical guide to building agents - a practical guide to building agents by OpenAI