L40S vs A100 for LLM Inference: Which GPU Has Lower Token Cost? Choosing by Concurrency, Context, and Precision

Let's start with the conclusion: When request concurrency is high, the model fits on a single card (7B/13B, or a 30B-class model after FP8 quantization), and the inference framework can enable FP8, L40S usually has a lower cost per million tokens. When concurrency is low, you need fast token-by-token output, contexts are very long, or you need to split an unquantized model above 70B across multiple cards, A100 80GB is the better fit—and for multi-GPU setups, prefer the SXM version.

The strengths of these two cards map to different stages of the inference process. Once you understand which stage your workload is mainly bottlenecked on, choosing a card becomes straightforward.

Inference has two stages, and each card excels in one

LLM inference can be broken down into two stages:

Prefill: processes the entire input in one pass, mainly compute-bound.
Decode: generates tokens one by one. Each generated token requires reading the model weights from VRAM again, so speed is mainly limited by memory bandwidth.

The key specs of the two cards are as follows:

A100: Ampere architecture, HBM2e memory. The 40GB version has about 1,555 GB/s bandwidth, and the 80GB version about 2,039 GB/s—more than twice that of the L40S. It does not support native FP8, and its BF16 peak compute is 312 TFLOPS.
L40S: Ada Lovelace architecture, 48GB GDDR6 memory, 864 GB/s bandwidth. Its 4th-gen Tensor Cores natively support FP8, with FP8 Tensor compute of about 733 TFLOPS.

So in single-user conversations where Batch Size is close to 1, the Decode stage is bandwidth-bound, and A100 clearly has lower per-token generation latency. As concurrency rises, a single weight read can serve more requests, and the bottleneck gradually shifts to compute—this is where the L40S's FP8 compute can shine.

Diagram showing that as Batch Size increases, the inference bottleneck shifts from memory bandwidth to compute, and L40S's cost per token gradually falls below A100's

Decide based on three variables

1. Concurrency

Generally speaking, once Batch reaches 8 or above, or after enabling Continuous Batching with frameworks like vLLM, the L40S's throughput advantage becomes more obvious. Combined with the L40S's typically lower hourly rental price than the A100, the cost per token becomes cheaper. If your service only has one or two users most of the time, the L40S's compute won't be fully utilized, and its bandwidth weakness will show—so it may not be cost-effective on a per-token basis.

2. Context length

The longer the context, the more VRAM the KV Cache takes up, and the more the Decode stage depends on bandwidth. For 32K, 64K, or even longer context windows, prioritize the A100 80GB, which is more generous in both memory capacity and bandwidth. The L40S's 48GB must be split between weights and KV Cache, so with long contexts plus high concurrency it's easy to hit the memory ceiling first.

3. Precision and model size

7B/13B models in FP16: the L40S can fit them on a single card, with plenty of room left for KV Cache.
30B-class models: the L40S needs FP8 or INT4 quantization to run on a single card. FP8 weights and FP8 KV Cache can roughly halve memory usage and roughly double effective throughput, offsetting some of the bandwidth disadvantage.
The A100 has no native FP8 hardware support, so this FP8 benefit isn't available on the A100. To run unquantized medium-to-large models, you mainly rely on the 80GB memory capacity.

Note that the L40S's FP8 advantage only takes effect when the inference framework enables the corresponding options, such as FP8 weights and KV Cache in vLLM and TensorRT-LLM. If you only run with the default FP16, the gap between the two cards will differ from expectations.

When multiple GPUs are needed, the interconnect matters a lot

If you don't quantize a model above 70B, it won't fit on a single card, and you need tensor parallelism (TP) to split it across 2, 4, or 8 cards. Tensor parallelism requires cross-GPU communication at every layer:

A100 SXM comes with 600 GB/s NVLink 3.0, so cross-GPU communication overhead is small, making it suitable for low-latency tensor parallelism.
L40S only has PCIe 4.0 (64 GB/s bidirectional) and does not support NVLink. In multi-GPU tensor parallelism, it tends to be slowed down by communication.

The L40S is better suited to running small and medium models independently on a single card, deploying multiple independent instances to share traffic, or using pipeline parallelism with less frequent communication. If you plan to use multiple A100s for tensor parallelism, before renting confirm whether the node is SXM or PCIe and whether it has NVLink—don't just look at the model name. For the specifics of multi-GPU splitting, see Llama model deployment in practice.

Quick reference

Your scenario	The card more likely to save money	Main reason
High-concurrency API service, 7B–13B models	L40S	After batching, compute-bound; lower rental price
30B-class models, FP8 quantization acceptable	L40S	FP8 saves memory and boosts throughput
Low-concurrency interaction, fast token-by-token output required	A100 80GB	Decode stage depends on bandwidth
Long contexts above 32K/64K	A100 80GB	KV Cache usage is large; both bandwidth and capacity are needed
Unquantized above 70B, multi-GPU tensor parallelism	A100 SXM	Low cross-GPU communication overhead via NVLink
Multiple small/medium models deployed independently	Multiple L40S instances	No cross-GPU interconnect dependency

The most reliable approach: test your own workload on each

The judgments in the table can only help narrow things down. Actual cost depends on the model, prompt length, output length, and concurrency curve. The most accurate approach is to run the same workload on both cards and calculate using the formula below:

Cost per million output tokens = hourly price ÷ (output tokens per second × 3600) × 1,000,000

Steps:

Keep test conditions consistent: model, precision, maximum context length, and test prompt set should all be the same. Run an extra FP8 set on the L40S and use BF16 on the A100 for a fair comparison.
Choose an inference image: in the NexGPU image templates, pick vLLM or TGI and deploy with one click, saving the time of configuring the environment yourself.
Stress test at realistic concurrency: run separately at your expected off-peak and peak concurrency, and record output tokens per second and per-token latency. Looking only at single-concurrency data will misjudge the L40S, while looking only at high-concurrency data will ignore the interactive experience.
Plug into the formula and compare: first confirm that latency meets business requirements, then choose the configuration with the lowest unit cost among those that meet requirements.

When checking rentable nodes on the pricing page, besides unit price, verify the memory spec (the A100 comes in 40GB and 80GB versions) and the interface type (SXM or PCIe). These two factors directly affect the judgments above.

Billing notes during testing

NexGPU is billed hourly and metered by the second, with no minimum spend, so short comparison tests don't cost much. A few things to note:

The unit price is locked in when you place the order, and it stays at that price until the instance is destroyed; the price won't change during testing.
Shutting down only stops compute billing; storage is still billed. After testing, if you want to keep the environment to continue another day, you can shut it down—but disk costs will keep accumulating.
Only destroying the instance stops all billing. Before destroying, export and save your stress-test results and configuration.

For billing details, see Are GPU instances still charged after shutdown?. If after testing you decide to deploy multiple GPUs long-term, or need to assess cluster scale, you can first organize your requirements using the enterprise GPU cluster consultation checklist, then contact sales.