Private LLM Chat · Self-hosted LLMs · AWS + Google Cloud

Your own ChatGPT, on your own GPU.

Open-weight models (Qwen, Llama, Gemma, DeepSeek) served with vLLM on Amazon SageMaker, several per GPU, with weights from S3, an optional no-internet VPC, and a GPU that switches itself off when idle. The Django and React app streams chat and handles documents, images and voice.
Status
Public · docs site with five guides
Licence
MIT
On GitHub
★ 1 · ⑂ 0
open-weight model families, several per GPU
4
GPUs running when idle
→ 0
guides, from a laptop to the full AWS stack
5
01

Why it exists

Sending company documents, customer data or private notes to a third-party AI service is often not an option. Running the model yourself is, but the path from an AWS account to a private chat app for a team is scattered across many blog posts.

This project is that path in one repository, from a demo on a laptop to a full deployment on AWS, with each step explained.

02

What it does

  • Private by design

    The model, the database and the files stay in your AWS account and region. The model endpoint has no public URL: only the app's IAM role can call it. Prompts are not logged.

  • A GPU that switches itself off

    Endpoints start when someone sends a message and are deleted after 30 idle minutes, so a forgotten GPU cannot run up a bill.

  • Any open model

    Presets for Qwen, Llama and Gemma, or any Hugging Face model that vLLM supports. The same app also talks to Ollama, LM Studio or any OpenAI-compatible server.

  • A full chat app

    Streaming answers, conversation history with search, Markdown with code and tables, instructions per user, and a model picker.

  • Documents, images, search and voice

    Attach PDFs and Word files, send images to vision models, pull text out of scans, search the web through a self-hosted SearXNG with citations, and dictate with Whisper.

  • Accounts with two-factor sign-in

    Email and password with verification, Sign in with Google, and two-factor codes from an authenticator app or by email.

03

How it works

  1. 01

    You press Send

    The Django API checks that the model's GPU is up, and starts it if it is not.

  2. 02

    The prompt is built

    The conversation history, your files and your instructions are put together and trimmed to fit the model's context window.

  3. 03

    The model answers

    The request goes to a SageMaker endpoint running vLLM, signed with IAM, and comes back as a stream.

  4. 04

    The answer reaches your browser

    The API relays each token to the Next.js app as it arrives.

fig. 01self-hosted stack
REACT APPstream · docs · images · voiceDjango APIstreaming rest · 2faWhisperOCRWeb searchSAGEMAKER · VLLMQwenLlamaGemmaDeepSeekGPUseveral models per gpuscale → 0 when idleweights from s3 · no-internet vpc optionS3 weightsofflinePrivate VPCoptionalModel Runnerlaptop · dockerTERRAFORM · ONE STACKalbcdnecssagemakerdatabasenetworkecremailcicddocs site · guides · gpu sizing · cost reference · MIT
04

Run it yourself

The first run needs no GPU. You need Docker, about 15 GB of free disk and 8 GB of memory. The first start asks which open model to download.

$ git clone https://github.com/codewithmuh/private-llm-chat-sagemaker-aws.git$ cd private-llm-chat-sagemaker-aws$ make up
  • Chat app: http://localhost:3000
  • Emails (Mailpit): http://localhost:8025
  • API and admin: http://localhost:8000/admin/
  • The guides then take you to a GPU on SageMaker and to the whole stack on AWS.
05

Stack

Model serving
vLLMAmazon SageMaker
Backend
Django 5.2Django REST Framework
Frontend
Next.js 16React 19TypeScript
Data
PostgreSQL (RDS)S3
Infrastructure
TerraformECS FargateCloudFrontALBSES
Local
Docker ComposeOllamaMailpit
06

Where it stands

Built

  • The chat app with accounts, documents, OCR, web search and voice input
  • Model providers for SageMaker, OpenAI-compatible servers and a mock for development
  • GPU control for each model: on demand, always on, off or manual
  • Terraform for the whole AWS stack, and CI for the backend, the frontend, Terraform and the Docker builds
  • Five guides, from a laptop to the full stack on AWS

Not yet

  • A Google Cloud Run port, which is early
  • A Bedrock provider, for comparisons

No production users are claimed. The Google Cloud Run port is early.

07

Questions

Do I need a GPU on my laptop?
No. The first two levels run on any machine. The GPU lives on SageMaker.
How much does it cost?
The app itself is roughly $80 to $90 a month on AWS. GPUs are billed by the hour while an endpoint exists: about $1.13 to $2.61 an hour for the single-GPU presets in us-east-1. With on-demand scaling you pay only for the hours used.
Why SageMaker and not a plain EC2 GPU?
Managed health checks, automatic restarts and IAM-only access to the model. The serving container is standard vLLM, so moving to EC2, EKS or your own hardware later is straightforward.
Why not Amazon Bedrock?
Bedrock is good for managed models, but you cannot run any open model you like with your own settings.
Is it production ready?
It is a secure base: sign-in with two-factor codes, private networking, IAM, encrypted storage, tests and CI. Before putting company data in it, review backups, email sending limits, a firewall and monitoring for your own needs.
→Work with me

Want Private LLM Chat running for your team, or something like it built?