What AI Can You Run on a Dedicated Server?

Part 7 of the AI series. This one assumes you know the basics from part 1. For a general look at dedicated servers, see the dedicated server guide.

A VPS tops out at small models. A dedicated server with 64 to 192 GB of RAM is a different animal: it can load models that no VPS can touch, run several models at once, and host a complete private AI stack on one machine. It still has no graphics card, so it is not fast, and this guide is honest about that. But with the right model choice it can be fast enough to be useful, and that is where the surprises are.

What is in this guide

• The one rule that explains AI speed on a CPU
• The surprise: models that are big but quick
• 1. A big model for patient, high-quality work
• 2. A fast mixture-of-experts model for daily chat
• 3. A small private AI service for a team
• 4. Several models, each for its job
• 5. A complete private AI stack on one box
• 6. Processing a whole archive
• 7. A big document search
• 8. A transcription factory
• 9. A coding assistant with a stronger model
• 10. A safe lab of isolated AI machines
• Choosing the hardware
• When you really need a GPU
• Does it pay for itself?
• Setting it up and keeping it safe

The one rule that explains AI speed on a CPU

To produce each word, the computer has to read the model's active numbers from memory. So the speed of a CPU model is limited mostly by memory bandwidth (how fast data moves from RAM), not by how many cores you have. A useful rule of thumb:

words per second is roughly memory bandwidth divided by the size of the data read for each word.

For example, a desktop-class processor with fast dual-channel DDR5 memory may move something on the order of 50 to 90 GB per second. A 4-bit 70B model is about 40 GB, so the ceiling is a word or two per second, and real life is a bit lower. A 4-bit 8B model is about 5 GB, so the ceiling is many times higher. These are rough upper limits, not promises, but they explain why model choice matters more than anything else on this kind of server.

The surprise: models that are big but quick

Some newer models are called mixture of experts. They hold a large amount of knowledge (for example, 20-35 billion parameters in total), but for each word they only use a small part of it (a few billion "active" parameters). Because speed depends on what is read for each word, not on the total size, these models can answer far faster than a dense model of the same size, as long as the whole thing fits in memory.

That is exactly the situation a big-RAM dedicated server is good at. Memory is plentiful, bandwidth is the limit, and a model that only touches a small slice per word makes the best use of it. Look for open mixture-of-experts models in the 20-35B range (the gpt-oss, Qwen and similar families have had them), and test them: for many people they are the best "feels fast enough" choice on a CPU server.

1. A big model for patient, high-quality work

What it is. A 30-70B dense model that gives noticeably better answers than the 7-8B size: better reasoning, better writing, fewer silly mistakes, better handling of long documents.

Why it suits this server. A 4-bit 70B model needs roughly 40 to 48 GB. A 64 GB server can hold it, and a 128 GB server holds it with plenty of room for a long context.

How to set it up. Use Ollama or llama.cpp as in part 3, and pull the large model. Give the server a few minutes to load it the first time. NVMe storage helps, because loading tens of gigabytes from a slow disk takes a long time.

Watch out for. It is slow: expect a word or two per second at best. Use it for jobs where you can wait, such as drafting a long report, reviewing a contract's wording or analysing a document overnight. For quick questions, keep a smaller model beside it.

2. A fast mixture-of-experts model for daily chat

What it is. A private assistant that answers at a comfortable pace, using a large mixture-of-experts model instead of a dense one.

Why it suits this server. You get the knowledge of a large model at the speed closer to a small one, which is the best trade on a CPU.

How to set it up. Pull a 20-35B mixture-of-experts model at 4-bit, and compare it with a dense model of similar size on your own questions. Try a few. The best one for you will probably change every few months.

Watch out for. Quality varies between models and tasks. Some of these models "think" for a long time before answering, which is slow on a CPU, so check that you can switch that off or lower it.

3. A small private AI service for a team

What it is. One server that a small team (a family, a small office, a club) shares, with their own accounts and histories, instead of everyone paying for a separate subscription.

Why it suits this server. Open WebUI handles accounts and permissions, and a fast mixture-of-experts model can serve short questions quickly. Memory is not the problem on this server.

How to set it up. Run Open WebUI, make accounts for each person, and turn off open sign-up. Put it behind HTTPS and a reverse proxy, and consider a VPN so it is not on the public internet at all.

Watch out for. A CPU handles one request at a time per model, so when several people ask at once they queue. Three to five light users are fine, and a larger group needs a bigger plan or a GPU. Agree on what may be pasted into it, and check the model's licence if you use it commercially.

4. Several models, each for its job

What it is. A fast small model for quick questions and classification, a large one for hard problems, an embedding model for search, a code model for programming, and a speech model, all installed and ready.

Why it suits this server. With 64-192 GB of RAM you can keep more than one model loaded, so there is no waiting while a model swaps in.

How to set it up. Allow several loaded models in Ollama (OLLAMA_MAX_LOADED_MODELS), and let the chat page pick per task. A gateway such as LiteLLM can route requests, sending easy jobs to the small model and hard ones to the big one.

Watch out for. Two models answering at once share the same memory bandwidth, so each gets slower. Memory fills quickly: add up the sizes before you load them.

5. A complete private AI stack on one box

What it is. Everything from the earlier guides, all on one machine: a chat page, a document library, speech in and out, OCR, automation and a few models.

Why it suits this server. The tools that squeeze a VPS fit easily here, and the data stays on a machine you control. See the tools map for the pieces.

How to set it up. Run each service in Docker on a private network, with one reverse proxy and one login in front. Keep a compose file in version control so you can rebuild it, and record every port.

Watch out for. Every extra service is more to patch and secure. A single dedicated server is a single point of failure, so back up the data and keep the setup written down.

6. Processing a whole archive

What it is. Running a model over a very large pile of material: years of emails, a library of PDFs, scanned records, thousands of support tickets. The output might be summaries, tags, extracted facts or a searchable index.

Why it suits this server. A fixed monthly price makes slow, steady work cheap. A job that would cost a lot through a pay-per-use service costs nothing extra once the server is rented, and a month of patient processing gets through a lot.

How to set it up. Write a resumable script that processes one item at a time, saves each result immediately and records which items are done. Test on a few hundred items and measure the error rate first.

Watch out for. Time it before you commit: take the seconds per item and multiply. If the answer is longer than a month, use a smaller model or a sample. Keep the raw outputs in case you want to change the prompt.

7. A big document search

What it is. Semantic search and question answering across a very large library: hundreds of thousands of pages, rather than hundreds.

Why it suits this server. Vector databases such as Qdrant hold millions of vectors in memory for fast search, and a large-RAM server can host the database, the embedding model and the answering model together.

How to set it up. Index in batches, store the original text alongside the vectors, and use a better embedding model than the tiny ones. Keep the number of passages sent to the answering model small, and show sources.

Watch out for. Embedding a huge library takes a long time on a CPU, so plan days and not hours. Poor chunking ruins results more than a weak model does, so spend time checking what the search finds.

8. A transcription factory

What it is. A service that turns piles of audio (a podcast archive, call recordings, lectures) into accurate text, using larger speech models than a VPS can handle.

Why it suits this server. Larger Whisper models are noticeably more accurate, especially with accents, noise and technical vocabulary. Many CPU cores let you run several jobs in parallel, and storage is plentiful.

How to set it up. Use faster-whisper with a larger model, run a queue (even a simple folder watcher), and split work so that several files run at once up to the number of cores. Add a language model afterwards for summaries.

Watch out for. Check the recording consent rules where you live. Run a few hours of audio first to measure throughput, because it varies a lot with the CPU.

9. A coding assistant with a stronger model

What it is. A private code assistant using a 30B-class code model, much better than the small ones at explaining unfamiliar code, writing larger pieces and finding bugs.

Why it suits this server. The bigger models fit. A mixture-of-experts code model keeps the speed tolerable, and the same server can run your build jobs and test runners.

How to set it up. Connect Continue or Aider to the server over a VPN or an SSH tunnel. Use a small fast model for completions and the big one for explanations and refactors.

Watch out for. The best paid models are still ahead on difficult tasks. Test your own typical jobs on both before deciding. Always review and run the tests on anything it writes.

10. A safe lab of isolated AI machines

What it is. Several virtual machines on one server, each running a different experiment: one for a risky agent that can run commands, one for a client project, one for trying new tools.

Why it suits this server. You can install Proxmox or KVM on a dedicated server and give each experiment its own machine, which a VPS cannot do. See the dedicated server guide for the virtualisation setup.

How to set it up. Give the model server its own machine and reach it from the others over a private network. Keep agents that run code in a disposable machine with no access to your real data or passwords.

Watch out for. Agents that act on their own can delete files, spend money and be tricked by text on the web pages they read. Isolation limits the damage, and snapshots let you undo it.

Choosing the hardware

1. RAM size first. It decides which models fit: about 48 GB for a 4-bit 70B, plus room for context, the system and other services. 64 GB is a floor for that, and 128 GB or more is comfortable.
2. Memory speed and channels second. It decides how fast the model answers. Newer DDR5 is faster than DDR4. Server-class processors with many memory channels are much faster for AI than desktop-class ones, but cost a lot more, so check what a listing really offers.
3. Cores matter less than you think for single-user chat, and more for parallel jobs such as transcription or batch work.
4. NVMe storage. Models are tens of gigabytes, so loading from fast storage saves minutes, and you will keep many models.
5. ECC memory is a bonus for a machine that runs unattended for months.
6. Network allowance. Downloading models uses a lot of bandwidth, but afterwards the traffic is small.

Here are the servers with the most memory in stock now:

When you really need a GPU

• You need answers from big models in seconds, not tens of seconds.
• Several people will use it at the same time.
• You want image or video generation.
• You want to fine-tune models beyond tiny ones.

The dedicated servers I looked at on the plans above are CPU servers. If a graphics card is a must, check what is offered before you order, and compare with specialist GPU rental.

Does it pay for itself?

A dedicated server costs the same whether you use it or not, and a pay-per-use service costs nothing when idle. It makes sense when you will use it heavily, such as batch jobs, long transcriptions or a team sharing it, or when privacy is the main reason. If you only chat occasionally, a pay-per-use service or a cheaper VPS is better value. Work it out with real numbers: your expected volume against the server's monthly price. It also helps that the machine does other jobs too, such as the ones in the dedicated server guide.

Setting it up and keeping it safe

1. Do the first-day checklist from the dedicated server guide: check the hardware, mirror the drives if you have two, secure SSH, set up backups.
2. Run Ollama and your tools in Docker as in part 3, with ports bound to 127.0.0.1.
3. Reach the tools through a VPN or an SSH tunnel, or an HTTPS reverse proxy with logins and sign-ups off.
4. Never expose the Ollama port directly. It has no login.
5. Test model speed with your real prompts before you build anything on it.
6. Back up your settings, document indexes and chat history. Models can be downloaded again.
7. Follow the security guide, and keep the system updated.

For smaller, cheaper options, see the 6-8 GB AI guide, the cheapest plans and the sales page.