SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

Lightweight Modular AI Routing Engine for Local LLMs — Run specialised experts efficiently on consumer GPUs using smart Mixture-of-Experts routing.

Giant MoE models on a single consumer GPU by streaming experts from SSD. CUDA fork of antirez/ds4: runs GLM-5.2 (743B), Tencent Hy3 (295B), and DeepSeek 4 Flash, with io_uring expert streaming, LFU host cache, cross-layer expert prefetch, and the first MTP speculative decoding for GLM-5.2.

Verified AI infrastructure for regulated deployment. UltraCompress (our wedge): near-lossless 5-bit compression with SHA-256-reproducible reconstruction - prove the model in production is the one you validated. 23 architectures (0.6B-405B), Hermes-3-405B @ 1.0066x. OpenAI-compatible API. pip install ultracompress

🦁 Local AI for Consumer GPUs — Run powerful LLMs on GTX 1060/1080. No cloud. No subscriptions. Built on llama.cpp + CUDA.

RAM-Backed MCP Memory Architecture for Consumer LLM Inference — 900K token context on 16GB VRAM

Arbitrary Numbers

Dynamic GPU Layer Swapping: Train large models on consumer GPUs with intelligent memory management

Surgical reasoning on consumer silicon. Hybrid SSM + causal memory architecture with entropy-gated System 1/2 dispatch, O(1) inference memory, and continual learning — designed for 16 GB VRAM.

AeloRu (Adaptive Elastic Learning with Orthogonal Robust Units) enables real-time, continuous learning on resource-constrained devices. Drop-in memory module for LLMs.

RAMP: RL-guided Adaptive Mixed-Precision quantization for GGUF models. Data-free sensitivity analysis, evolutionary search, per-tensor type optimization. Produces hardware-optimized GGUF for consumer GPUs.

Self-hosted LLM chat client with streaming UI for vLLM servers. Run Mistral-24B locally on RTX 4090/3090. Privacy-focused ChatGPT alternative for homelab/gaming PCs. Python/Rich terminal UI.

Reproducible local inference for FLUX.1 [schnell] on 8 GB Turing GPUs (RTX 2070). Sequential CPU offload, fp16 compute, mock-tested pipeline init - portfolio piece for a neuroscience to ML transition.

Adaptive hot/cold MoE expert cache for llama.cpp: keep the hottest experts in VRAM to speed up decode when offloading to RAM on a consumer GPU. Measured +26% (Qwen3-Coder-Next 80B) and +31.6% (gpt-oss-120B) on one RTX 5090.

Researching local AI image editing on affordable consumer hardware: models, ComfyUI workflows, benchmarks, and real outputs on an RTX 5060 Ti 16GB.

A comprehensive, modular framework for fine-tuning Stable Diffusion 3.5 models using LoRA (Low-Rank Adaptation). Create custom AI image generators tailored to your artistic style, objects, or concepts with memory-efficient training on consumer GPUs.

Tiny GPT-style training and local inference demo for consumer hardware.

Technical notes on building a local-first AI coding assistant with local LLMs, Ollama, SwiftUI, and consumer GPU constraints.

Command-line interface for Kimari Local AI — LLM inference gateway for consumer GPUs