Custom LLM: how to train an LLM on your own data

A custom LLM is a language model that knows your company: your products, your terms, your rules and the way your team writes. Almost no company builds one from scratch. You start from an open-weight model, one whose trained files are published so you can download and run them, and you fine-tune it on your own records.

Done well, the result runs on your own servers or private cloud, and your company owns it. This guide is for operations leads and owners who want to know what that takes, in plain terms: when fine-tuning fits and when retrieval is enough, how to prepare data, how to test, how to keep it private, where it runs and what drives the cost. It is the work behind our LLM development service.

What is a custom LLM?

A large language model (LLM) writes by prediction. NIST's Generative AI Profile puts it simply: these models produce outputs that approximate the statistical distribution of their training data, predicting the next word in a sentence. A general model has learned general patterns. A custom one has been adapted to yours.

There are three common ways to adapt a model:

• Instructions: a written brief the model reads every time. Quick to set up, but limited.
• Retrieval: the model looks up passages in your documents when a question comes in, then answers from them.
• Fine-tuning: you train the model further on your own examples, so what it learns lives in its weights, the files that hold what it has learned.

Training a model from nothing, called pre-training, is a far bigger project that few companies need.

Fine-tuning or retrieval: which one fits?

They solve different problems, and many custom models use both.

	Retrieval	Fine-tuning
What changes	The model reads your documents at question time	The model's weights learn from your examples
Best for	Facts that change, answers that must cite a source	Your terms, formats, tasks and tone
Keeping it current	Add or remove a document	Retrain on new examples
Data you need	Organized documents	Clean worked examples: a request and the right answer
Main thing to watch	Finding the wrong passage, or instructions hidden in a document	Learning mistakes or private details from the data

Choose retrieval when the facts change often, like prices, policies or stock levels, or when every answer must point to its source. Choose fine-tuning when the model needs to work the way your team works: use your product names correctly, write in your format, sort requests into your categories and follow your rules without a page of instructions each time.

The common pattern is a fine-tuned model that also retrieves. The tuning teaches it how your company works; retrieval hands it today's facts and the document to cite. NIST's profile names both as ways to adapt a pre-trained model, and suggests you document which one you used and re-assess the risks after either.

If retrieval alone covers what you need, that is a different build; see AI systems development.

How to train an LLM on your own data, step by step

1. Pick one job. Start with a question your team answers again and again. Write down what a good answer looks like and who checks it.
2. Gather the data. Documents, emails, help desk tickets, standard procedures and CRM history are the usual sources. Record where each set came from. NIST suggests documenting training data sources so you can trace their origin, and checking whether your use of proprietary or sensitive data is consistent with applicable laws.
3. Clean it. Remove duplicates, a check NIST lists on its own. Drop outdated versions: the AI Risk Management Framework warns that training data can become stale or detached from its intended context. Strip email signatures, long quoted threads and broken text. Mask personal details the job doesn't need. Write the cleaning rules down so the next round follows them.
4. Turn records into examples. Each example is a real request and the answer your best people would give. The people who do the job write or approve them. Cover the full range of cases, not only the common ones; NIST flags training data imbalances as a source of harmful bias.
5. Set aside a test set first. Hold back real cases the model never sees in training. They are how you will judge it.
6. Fine-tune an open-weight model. Pick a size that fits the job and the hardware. NIST suggests looking at a model's baseline benchmark results when you choose one to fine-tune.
7. Test, then pilot with approvals on.
8. Install, connect and monitor. Put it on your systems, connect it to email, CRM or help desk, and watch how it does on live work.

How do you test a custom LLM before it goes live?

The NIST framework says accuracy measurements should always be paired with clearly defined and realistic test sets that represent the conditions of expected use. In practice:

• Test on your own held-back cases. Benchmarks help you choose a model, but NIST's profile notes their results may not carry over to real-world use.
• Have the people who do the job grade the answers: right, wrong, or right but needs editing. Everyone else who uses it needs the basics too: see AI training for employees.
• Compare against today. Run the same cases through the untuned model, or your current process, so you can see what tuning added.
• Check its sources. If it cites documents, confirm they say what the answer claims. NIST suggests reviewing sources and citations in outputs before launch and after.
• Try to make it leak. Ask for details it should never reveal. NIST suggests testing whether a model outputs samples of its training data.
• Don't decide on a few good demos. The profile warns against judging a model from narrow, anecdotal assessments.

Keep testing after launch. The framework expects a model to be monitored in production, and notes that these systems may need more frequent maintenance as data and conditions drift. Anything that reaches a customer or moves money should wait for a person's approval.

How do you keep your data private?

Build privacy in from the first step.

• Know what models can leak. NIST's profile notes that models may leak, generate or correctly infer sensitive information, a problem called data memorization, even for data present in only a few training samples. Remove or mask what the job doesn't need before training.
• Minimize. The framework points to data minimizing methods such as de-identification and aggregation.
• Protect the weights. A model trained on your records carries some of what it learned from them. NIST counts the confidentiality of training data and model weights as part of information security. Treat the files like the records they came from.
• Watch what goes in. NIST describes indirect prompt injection, where an attacker plants instructions in data a system is likely to retrieve. Limit retrieval to sources you control, and check new documents before they are added.
• Keep it inside. A model on your own servers or private cloud means prompts and answers stay inside your network.

Where does it run, and what drives the cost?

A custom LLM can run on servers in your office or data center, or in a private cloud account your company owns. NIST's profile calls training and running these models resource-intensive, and notes that energy use varies with the activity, the hardware and the type of task.

The cost drivers, in plain terms:

• Model size. Bigger models need more memory and more expensive hardware. Choose the smallest one that passes your tests.
• How many people use it, and how often. More use means more hardware.
• How much data needs cleaning. Messy records can take more work than the training itself.
• How often you retrain. Put fast-changing facts in retrieval, and you retrain less.
• Testing and monitoring. Ongoing checks are part of the running cost.

A cloud AI tool is billed per seat or per use, so the bill grows with use. With your own model, the cost sits mostly in hardware and upkeep, with no per-use fee to an AI vendor. For the wider decision, see our guide to build vs. buy software.

What should you own at the end?

If you pay for a custom LLM, you should be handed more than a login:

• The trained model, weights included, on your systems.
• Your cleaned training data and the rules used to clean it.
• The test set and how the model did on it.
• A runbook: how it is installed and connected, and how to keep it running.

Also check the license of the open-weight model you start from, since it sets how you may use the result.

How Raileon builds a custom LLM

Raileon fine-tunes an open-weight model on your company's own data. It runs on your own servers or private cloud, and your company owns it, weights included. We never use your data to train AI models, unless you hire us to train your own model. Then it is trained only for you, and our security page has the full policy.

It starts with a free 20-minute call. Then a written proposal names the data we'd train on, the model, the hardware it needs and who approves its work, with the exact price. Builds are priced by scope. Monthly care (hosting, monitoring, fixes and updates) starts from $1,999 a month. Either side can end the service on 30 days' notice, and the model stays yours. See LLM development for how an engagement runs.

FAQ

What is a custom LLM?

A custom LLM is a language model adapted to one company's data and work. Most are built by fine-tuning an open-weight model on the company's own records, often with retrieval for facts that change.

Can you train an LLM on your own data without sending it to a vendor?

Yes. An open-weight model can be fine-tuned and run on your own servers or private cloud, so your records and the trained model stay on your systems.

How much data do you need to fine-tune a model?

It depends on the job. What matters most is clean, varied examples approved by the people who do the work, plus a held-back test set that shows whether it works.

Is fine-tuning better than retrieval?

Neither is better on its own. Retrieval suits facts that change and answers that need a source. Fine-tuning suits your terms, formats and tasks. Many custom models use both.

Find out if a private model fits

Bring one question your team answers again and again. On a free 20-minute call, we'll tell you whether your own model could handle it, what data it would need and what it wouldn't do.

Book a free 20-minute call