Roles: engineering leaders, security and risk owners, finance leaders. Capabilities: any generative or predictive model the organization runs or calls. Decision: who operates the model, and what that obliges each party to do. Linked chapters: model selection, retrieval, knowledge retrieval, build, buy, or configure, vendor diligence.
The decision
A team that has chosen what a model must do still has to choose where it runs and who looks after it. The options range from calling a provider's hosted service to running a model's weights on infrastructure the organization controls. The choice is sometimes framed as a matter of principle, open against closed. This chapter treats it as a requirements decision: which arrangement satisfies the workflow's data-handling, isolation, change-control, availability, and cost requirements, and which party then carries each operational responsibility.
The unit of analysis is the operating arrangement for one workflow, not a company-wide stance. A firm might call a hosted service for drafting proposal language and run a model inside its own environment for a knowledge assistant whose engagement files are bound by client confidentiality. Both choices can be right.
This chapter describes arrangements and responsibilities. It contains no provider price, no performance comparison, and no survey of which models are released under which terms (the two licenses quoted below are examples read at source). Version-specific details change often, so check them at source for the specific model and version, and read every license for that model and version. Recommended practices are proposed, labeled as such, and untested in any deployment by this guide.
Four arrangements
For your technical lead
The arrangements below are a working vocabulary, not a provider taxonomy; providers package these options differently and some products combine them.
| Arrangement | Who serves, scales, and patches | Who controls data flow and retention | Who decides when the model changes | Isolation available |
|---|---|---|---|---|
| Hosted service through an API | Provider | Governed by the provider's terms and your account settings | Provider publishes versions; you choose which identifier to call, within what the provider offers | Logical separation within the provider's service |
| Provider-managed dedicated or private deployment | Provider, on capacity reserved for you | Your contract and configuration, within the provider's environment | Provider and you, under the contract | Dedicated capacity; network controls depend on the offering |
| Open-weight model on a cloud platform you operate | You, using the platform's compute and managed services | You, within the cloud platform's terms | You | Your own network and account boundaries |
| Open-weight model on infrastructure you run | You, including hardware | You | You | Up to full separation from external networks |
Two points follow from the table. First, the further down the table, the more responsibility moves to your team; isolation and control are bought with operational work. Second, even the hosted arrangement leaves real decisions with you: which model identifier to call, what data to send, what to retain in your own logs, and when to re-evaluate.
Hosted models can change in ways you do not control
A provider controls the serving stack behind a hosted model. One provider's documentation states that, for its recent generations, a dateless model identifier maps to a single fixed snapshot and updated versions ship under new identifiers, while serving infrastructure can still change and occasionally produce minor differences in observable behavior. Anthropic, model IDs and versions That is one provider's convention and others differ. The practical consequence is the same: record the exact identifier your workflow calls, and treat a version change, a deprecation notice, or an unexplained shift in your monitoring as a trigger to re-run your evaluation. The model-selection chapter lists re-evaluation triggers.
Running weights yourself removes provider-initiated change but not change itself. A new release, a serving-runtime upgrade, a different precision setting, or a library update can alter behavior, and each is now your decision and your test.
"Open weight" is not the same as "open source"
Releasing a model's weights lets others run and modify that model. It does not, by itself, grant the freedoms associated with open-source software, and the terms attached to weights differ from model to model.
The Open Source Initiative's definition of Open Source AI, version 1.0, requires that a system be available under terms granting freedom to use, study, modify, and share it, and that the preferred form for modification be available: detailed information about the training data, the complete code used to train and run the system, and the parameters, each under OSI-approved terms. Open Source Initiative, Open Source AI Definition A model released with weights but without training-data information or training code does not meet that definition. This guide does not assess whether any particular model meets it.
Licenses attached to weights vary in what they permit and require. Two examples, read at source:
- The Apache License 2.0 grants each recipient a perpetual, worldwide, royalty-free copyright license and a patent license, subject to conditions including notice and attribution requirements. Apache License 2.0 Whether a given model's weights are released under it must be checked in that model's own repository.
- The Llama 3.1 Community License requires distributors to include the agreement and display "Built with Llama," requires distributed models trained or fine-tuned with Llama materials or outputs to begin their names with "Llama," incorporates an acceptable use policy by reference, and requires licensees whose products exceeded 700 million monthly active users in the month before the release date to request a separate license. Llama 3.1 Community License
These are two licenses at specific versions, cited to show that terms carry conditions, not to characterize open-weight licensing generally. Proposed practice: record the license, its version, and any acceptable use policy for every model in the configuration record, and have whoever owns legal obligations review them before the model is used in a product or distributed. This chapter is not legal advice.
Start from requirements
The model-selection chapter separates requirements that are pass or fail from trade-offs. Deployment requirements belong mostly in the first group. The table below lists questions that decide the arrangement; it proposes how to read the answers, not a rule.
| Requirement | Question to answer | What the answer implies |
|---|---|---|
| Data handling | May this workflow's data leave your environment, and under what terms? | If not, hosted APIs are out unless a contract and configuration satisfy the obligation |
| Isolation | Must the system run without external network access? | Points to running weights on infrastructure you control |
| Retention | Who may retain prompts, outputs, and logs, and for how long? | Check provider terms and your own logging; both count |
| Change control | Must the model stay fixed until you approve a change? | Pinned identifiers may suffice within the provider's deprecation window; a model that must stay fixed beyond it needs self-operation |
| Availability | What happens to the workflow if the model is unavailable? | Every arrangement needs a fallback; self-operation adds your own uptime obligations |
| Latency | What response time does the workflow need at its peaks? | Determines capacity you must provision if you serve the model |
| Customization | Does the workflow need fine-tuning or a model the provider does not offer? | May require weights you can modify, subject to the license |
| Staff | Who will serve, monitor, patch, and evaluate the model? | If no one can, self-operation is not yet a real option |
A requirement such as "client engagement files may not leave our environment" can settle the question on its own, as the knowledge-retrieval chapter's confidentiality constraints might for some firms. A preference such as "we would rather not depend on a vendor" is a trade-off, to be weighed against the operational work it brings.
What you take on when you run the weights
For your technical lead
Running a model is running a production service. The responsibilities below exist under every arrangement; the question is who carries them.
| Responsibility | Hosted service | Weights you operate |
|---|---|---|
| Serving and scaling | Provider | Your team: inference runtime, hardware or cloud capacity, autoscaling |
| Capacity for peaks | Provider, within rate limits you negotiate | Your team; idle capacity costs money, and too little degrades the workflow |
| Security patching | Provider for its stack; you for your integration | Your team for the runtime, libraries, operating system, and hardware |
| Monitoring | Your workflow metrics; provider status for the service | Everything: latency, errors, utilization, output quality |
| Evaluating each release | You decide when to adopt a new identifier | You decide and test every new model, runtime, and configuration |
| Incident response | Shared: provider for the service, you for the workflow | Your team, end to end |
| Model provenance | Provider's assurance under contract | Your verification of where the weights came from |
Provenance and the supply chain
Downloaded weights are a dependency like any other software, with a difference: their behavior cannot be fully inspected by reading them. OWASP's guidance on LLM supply-chain risk notes that model cards and documentation offer no guarantee of a model's origin, lists vulnerable pre-trained models and licensing among the risks, and recommends vetting suppliers, maintaining a signed software or AI bill of materials, using models from verifiable sources with integrity checks such as signing and file hashes, and red teaming models for the intended use. OWASP LLM03:2025 That is community guidance, not a measurement of how frequently supply-chain attacks occur.
Proposed controls: download weights only from the publisher's verified source; record and check file hashes or signatures; keep the weights, runtime, and libraries in the same inventory as other production software; and run the workflow's evaluation set, including adversarial cases, before first use and after every change.
Running the weights does not remove other risks
Self-operation changes who can see the data; it does not make the model's outputs safer. Prompt injection, the use of untrusted input to make a system act contrary to its design, applies wherever a model reads documents, tickets, or web content, whoever hosts it, and so does the need for permission-filtered retrieval as the retrieval chapter describes. Reviewers can over-trust outputs from a system the team built as readily as from one it bought; the authority chapter lists the signs.
Failure modes and controls
For your technical lead
| Failure | How it shows | Detection | Proposed control | Owner |
|---|---|---|---|---|
| Unannounced behavior change | Output quality or format shifts without a code change | Monitoring of quality signals and output schema | Record exact identifiers; re-run evaluation on any change | Engineering |
| Data sent where it should not go | Restricted content in a hosted request or third-party log | Audit of request content and retention settings | Classify data per workflow; block restricted classes from hosted calls | Security and the data owner |
| Under-provisioned self-hosting | Latency spikes or timeouts at peaks | Latency percentiles and queue depth | Size for peaks; define degraded mode and fallback | Engineering |
| Unpatched runtime | Known vulnerability in serving stack | Dependency scanning against the inventory | Patch policy covering runtime, libraries, and drivers | Engineering and security |
| Weights of unverified origin | Model file differs from the publisher's | Hash or signature check | Verified download source; integrity check in deployment | Engineering |
| License condition missed | Distribution or naming that breaches terms | License review at selection and before release | License recorded in the configuration record | Legal owner |
| Configuration drift | Self-hosted results differ from evaluation | Comparison of production configuration with evaluated one | Treat runtime, precision, and settings as part of the configuration | Engineering |
Evaluate every arrangement the same way
For your technical lead
The serving configuration is part of what is evaluated. The same weights served with different numeric precision, context limits, or sampling settings can behave differently, so a result from a provider's hosted version of a model does not carry over automatically to your own deployment of its weights. Proposed practice, extending the model-selection configuration record:
- Record the deployment arrangement, serving runtime and version, numeric precision, hardware or instance type, and generation settings alongside the model identifier.
- Run the same held-out evaluation set, including challenge cases and scope tests, against each candidate arrangement.
- Measure latency at the workflow's realistic peak, not only on a single request.
- Repeat the evaluation after any change to weights, runtime, or settings, and keep the results with the decision record.
No arrangement in this guide has been benchmarked, and no comparison of hosted and self-operated quality is claimed.
Cost and latency
The cost structures differ in shape. Pay-per-use API access charges for what is consumed, so cost rises with volume and idle time costs nothing. Reserved or dedicated capacity, whether provider-managed or self-operated, costs money whether or not it is used, and must be sized for peaks; autoscaling on a cloud platform sits between the two, scaling down when idle at the price of start-up delay. Whatever the arrangement, the staff time to serve, patch, monitor, and evaluate weights you operate continues regardless of volume. Which is cheaper for a given workflow depends on volume, how evenly it is spread, the capacity a peak requires, and the people already available. This guide does not estimate where any crossover falls; the inputs are too specific to the workflow and to prices that change.
Count the continuing work in the comparison. The economics chapter separates recurring cash costs from capacity consumed by existing staff; an engineer who spends a quarter of their time keeping a self-hosted model healthy is a recurring cost even if no new invoice arrives. Setup for self-operation includes infrastructure, security review, and building the evaluation and monitoring that a hosted service partly provides.
Latency depends on the serving stack and on where the model runs relative to the workflow. A self-hosted model inside the same network can avoid external round trips; a model on under-provisioned capacity can be slower than a hosted one. Measure rather than assume.
When hosted is simpler
Prefer a hosted service when the workflow's data may be sent under acceptable terms, when no one on the team is ready to operate a model in production, when volume is low or irregular, and when the provider's change notices and pinned identifiers give enough control. Several of the guide's workflow chapters, including support replies and meeting follow-up, can be piloted on a hosted service if the data terms allow, with the evaluation and permission controls those chapters describe.
Consider operating weights yourself when a pass-or-fail requirement demands it: data that may not leave the environment, a need for full isolation, a model that must stay fixed for a long validation period, or a modification the license permits and no provider offers. Start the operational work before the pilot, not after: a pilot run on a hosted service and then moved to self-operation needs a second evaluation.
A provider-managed dedicated deployment can satisfy some isolation and change-control requirements without self-operation. Read its terms with the same care; "dedicated" describes capacity, and what it implies for data handling and access is set by the contract.
People and skills
Self-operation needs people who can run inference infrastructure, size capacity, patch and monitor production services, and build and maintain an evaluation harness. The engineering leaders guide describes these as platform responsibilities. A hosted arrangement still needs someone who understands the provider's data terms, monitors behavior, and owns re-evaluation. Security and legal owners need to review data flows and licenses in either case; finance needs the continuing costs in the plan, as the finance leaders guide describes.
What external research establishes
For your analyst
The sources cited here are definitions, license texts, vendor documentation, and community security guidance. The Open Source Initiative's definition explains why released weights alone are not open source in its sense; the Apache 2.0 and Llama 3.1 texts show that model licenses carry different conditions; OWASP describes supply-chain risks with recommended controls (injection risks are cited in the retrieval chapter); one provider's documentation describes how its model identifiers relate to snapshots. None of these sources compares the quality, cost, or reliability of hosted and self-operated deployment, and this chapter makes no such claim.
Evidence still needed and related reading
Obtain an attributable account from a team that has operated an open-weight model in production: staffing, incidents, evaluation cadence, and continuing cost against a hosted alternative for the same workflow. Measure, for a specific workflow, latency at peak and output quality across arrangements using the same evaluation set. Have counsel review the licenses of any model under consideration. No team has been interviewed and no deployment measured for this chapter.
Continue with model selection, retrieval, build, buy, or configure, vendor diligence, and knowledge retrieval.