i-serve

Running an AI model on a rented GPU is a trade between speed, how many people you serve at once, and money. Describe your situation, and the page works out the trade.

I serve on one that costs me an hour. A typical prompt is tokens and the answer ; about of each prompt repeats something the card has already seen.

Change anything dotted. Every answer below follows. A token is about three quarters of an English word.

More assumptions, and the numbers behind the answers
KV cache dtype FP8 stores each seat's memory in half the bytes: twice the room, twice the time limit, the same gap between them
1 · Speed

How fast should each word appear?

People reading a chat notice anything slower than about 50 ms per token. A batch job only needs its deadline, so it can trade speed for seats.

ms per token
the card can hold
—

you can safely promise
—

first word after
—

    2 · Cost

    What does a million tokens cost at that speed?

    Only tokens that arrived on time count. A faster promise means fewer people share the hourly bill, so each token costs more.

    for you, today
    —

    hardware alone
    —

    the cheapest move
    —

      3 · Trouble

      It got slow. Where do you look first?

      Answer what your dashboard shows; don't know is fine. The page names the one cause that fits your operating point, and the knob for it.

      Are people queuing up? row 3 · waiting
      Is every seat taken? row 3 · running = max_num_seqs
      Is the first word late? row 2 · TTFT p99 against its floor
      Are the words slow? row 2 · TPOT p99 against the promise
      I have the numbers, let me type them

      Leave a field empty for No data. On kind that is most of them: the stub exports the two queue gauges and no histogram.