Show HN: Serve 100 Large AI models on a single GPU with low impact to TTFT

1 comments

• billconan a day ago ago

  can you hot swap a portion of an ai model, if my gpu is not large enough to hold the entire model? so that I can run half model first and load the other half.