Private LLM Chat · Self-hosted LLMs · AWS + Google Cloud
Your own ChatGPT, on your own GPU.
- Status
- Public · docs site with five guides
- Licence
- MIT
- On GitHub
- ★ 1 · ⑂ 0
- open-weight model families, several per GPU
- 4
- GPUs running when idle
- → 0
- guides, from a laptop to the full AWS stack
- 5
Why it exists
Sending company documents, customer data or private notes to a third-party AI service is often not an option. Running the model yourself is, but the path from an AWS account to a private chat app for a team is scattered across many blog posts.
This project is that path in one repository, from a demo on a laptop to a full deployment on AWS, with each step explained.
What it does
Private by design
The model, the database and the files stay in your AWS account and region. The model endpoint has no public URL: only the app's IAM role can call it. Prompts are not logged.
A GPU that switches itself off
Endpoints start when someone sends a message and are deleted after 30 idle minutes, so a forgotten GPU cannot run up a bill.
Any open model
Presets for Qwen, Llama and Gemma, or any Hugging Face model that vLLM supports. The same app also talks to Ollama, LM Studio or any OpenAI-compatible server.
A full chat app
Streaming answers, conversation history with search, Markdown with code and tables, instructions per user, and a model picker.
Documents, images, search and voice
Attach PDFs and Word files, send images to vision models, pull text out of scans, search the web through a self-hosted SearXNG with citations, and dictate with Whisper.
Accounts with two-factor sign-in
Email and password with verification, Sign in with Google, and two-factor codes from an authenticator app or by email.
How it works
- 01
You press Send
The Django API checks that the model's GPU is up, and starts it if it is not.
- 02
The prompt is built
The conversation history, your files and your instructions are put together and trimmed to fit the model's context window.
- 03
The model answers
The request goes to a SageMaker endpoint running vLLM, signed with IAM, and comes back as a stream.
- 04
The answer reaches your browser
The API relays each token to the Next.js app as it arrives.
Run it yourself
The first run needs no GPU. You need Docker, about 15 GB of free disk and 8 GB of memory. The first start asks which open model to download.
$ git clone https://github.com/codewithmuh/private-llm-chat-sagemaker-aws.git$ cd private-llm-chat-sagemaker-aws$ make up- Chat app: http://localhost:3000
- Emails (Mailpit): http://localhost:8025
- API and admin: http://localhost:8000/admin/
- The guides then take you to a GPU on SageMaker and to the whole stack on AWS.
Stack
Where it stands
Built
- The chat app with accounts, documents, OCR, web search and voice input
- Model providers for SageMaker, OpenAI-compatible servers and a mock for development
- GPU control for each model: on demand, always on, off or manual
- Terraform for the whole AWS stack, and CI for the backend, the frontend, Terraform and the Docker builds
- Five guides, from a laptop to the full stack on AWS
Not yet
- A Google Cloud Run port, which is early
- A Bedrock provider, for comparisons
No production users are claimed. The Google Cloud Run port is early.
Questions
- Do I need a GPU on my laptop?
- No. The first two levels run on any machine. The GPU lives on SageMaker.
- How much does it cost?
- The app itself is roughly $80 to $90 a month on AWS. GPUs are billed by the hour while an endpoint exists: about $1.13 to $2.61 an hour for the single-GPU presets in us-east-1. With on-demand scaling you pay only for the hours used.
- Why SageMaker and not a plain EC2 GPU?
- Managed health checks, automatic restarts and IAM-only access to the model. The serving container is standard vLLM, so moving to EC2, EKS or your own hardware later is straightforward.
- Why not Amazon Bedrock?
- Bedrock is good for managed models, but you cannot run any open model you like with your own settings.
- Is it production ready?
- It is a secure base: sign-in with two-factor codes, private networking, IAM, encrypted storage, tests and CI. Before putting company data in it, review backups, email sending limits, a firewall and monitoring for your own needs.
Want Private LLM Chat running for your team, or something like it built?