How you can Run A number of AI Workloads on a Single GPU

Introduction: What’s GPU Fractioning?

GPUs are in extraordinarily excessive demand proper now, particularly with the fast development of AI workloads throughout industries. Environment friendly useful resource utilization is extra essential than ever, and GPU fractioning is without doubt one of the best methods to attain it.

GPU fractioning is the method of dividing a single bodily GPU into a number of logical models, permitting a number of workloads to run concurrently on the identical {hardware}. This maximizes {hardware} utilization, lowers operational prices, and allows groups to run numerous AI duties on a single GPU.

On this weblog publish, we’ll cowl what GPU fractioning is, discover technical approaches like TimeSlicing and Nvidia MIG, talk about why you want GPU fractioning, and clarify how Clarifai Compute Orchestration handles all of the backend complexity for you. This makes it simple to deploy and scale a number of workloads throughout any infrastructure.

Now that we’ve got a high-level understanding of what GPU fractioning is and why it issues, let’s dive into why it’s important in real-world situations.

Why GPU Fractioning Is Important

In lots of real-world situations, AI workloads are light-weight in nature, typically requiring solely 2-3 GB of VRAM whereas nonetheless benefiting from GPU acceleration. GPU fractioning allows:

• Price Effectivity: Run a number of duties on a single GPU, considerably decreasing {hardware} prices.

• Higher Utilization: Prevents under-utilization of pricy GPU assets by filling idle cycles with extra workloads.

• Scalability: Simply scale the variety of concurrent jobs, with some setups permitting 2 to eight jobs on a single GPU.

• Flexibility: Helps various workloads, from inference and mannequin coaching to information evaluation, on one piece of {hardware}.

These advantages make fractional GPUs significantly enticing for startups and analysis labs, the place maximizing each greenback and each compute cycle is vital. Within the subsequent part, we’ll take a better take a look at the commonest methods used to implement GPU fractioning in observe.

Deep Dive: Widespread Strategies for Fractioning GPUs

These are probably the most extensively used, low-level approaches to fractional GPU allocation. Whereas they provide efficient management, they typically require guide setup, hardware-specific configurations, and cautious useful resource administration to stop conflicts or efficiency degradation.

1. TimeSlicing

TimeSlicing is a software-level method that permits a number of workloads to share a single GPU by allocating time-based slices. The GPU is nearly divided into a hard and fast variety of slices, and every workload is assigned a portion primarily based on what number of slices it receives.

For instance, if a GPU is split into 20 slices:

• Workload A: Allotted 4 slices → 0.2 GPU

• Workload B: Allotted 10 slices → 0.5 GPU

• Workload C: Allotted 6 slices → 0.3 GPU

This provides every workload a proportional share of compute and reminiscence, however the system doesn’t implement these limits on the {hardware} degree. The GPU scheduler merely time-shares entry amongst processes primarily based on these allocations.

Vital traits:

• No precise isolation: All workloads run on the identical GPU with no assured separation. On a 24GB GPU, for example, Workload A ought to keep beneath 4.8GB of VRAM, Workload B beneath 12GB, and Workload C beneath 7.2GB. If any workload exceeds its anticipated utilization, it could possibly crash others.

• Shared compute with context switching: If one workload is idle, others can quickly make the most of extra compute, however that is opportunistic and never enforced.

• Excessive danger of interference: Since enforcement is guide, incorrect reminiscence assumptions can result in instability.

2. MIG (Multi-Occasion GPU)

MIG is a {hardware} function accessible on NVIDIA A100 and H100 GPUs that permits a single GPU to be cut up into remoted situations. Every MIG occasion has devoted compute cores, reminiscence, and scheduling assets, offering predictable efficiency and strict isolation.

MIG situations are primarily based on predefined profiles, which decide the quantity of reminiscence and compute allotted to every slice. For instance, a 40GB A100 GPU could be divided into:

• 4 situations utilizing the 2g.10gb profile, every with round 10GB VRAM

• 7 smaller situations utilizing the 1g.5gb profile, every with about 5GB VRAM

Every profile represents a hard and fast unit of GPU assets, and workloads can solely use one occasion at a time. You can not mix two profiles to offer a workload extra compute or reminiscence. Whereas MIG provides strict isolation and dependable efficiency, it lacks the pliability to share or dynamically shift assets between workloads.

Key traits of MIG:

• Robust isolation: Every workload runs in its personal devoted area, with no danger of crashing or affecting others.

• Fastened configuration: You will need to select from a set of predefined occasion sizes.

• No dynamic sharing: Not like TimeSlicing, unused compute or reminiscence in a single occasion can’t be borrowed by one other.

• Restricted {hardware} assist: MIG is barely accessible on sure information center-grade GPUs and requires specialised setup.

How Compute Orchestration Simplifies GPU Fractioning

One of many largest challenges in GPU fractioning is managing the complexity of establishing compute clusters, allocating slices of GPU assets, and dynamically scaling workloads as demand modifications. Clarifai’s Compute Orchestration handles all of this for you within the background. You don’t must handle infrastructure or tune useful resource settings manually. The platform takes care of every thing, so you may give attention to constructing and delivery fashions.

Fairly than counting on static slicing or hardware-level isolation, Clarifai makes use of clever time slicing and customized scheduling on the orchestration layer. Mannequin runner pods are positioned throughout GPU nodes primarily based on their GPU reminiscence requests, making certain that the overall reminiscence utilization on a node by no means exceeds its bodily GPU capability.

Let’s say you will have two fashions deployed on a single NVIDIA L40S GPU. One is a big language mannequin for chat, and the opposite is a imaginative and prescient mannequin for picture tagging. As a substitute of spinning up separate machines or configuring advanced useful resource boundaries, Clarifai mechanically manages GPU reminiscence and compute. If the imaginative and prescient mannequin is idle, extra assets are allotted to the language mannequin. When each are energetic, the system dynamically balances utilization to make sure each run easily with out interference.

This method brings a number of benefits:

• Sensible scheduling that adapts to workload wants and GPU availability

• Automated useful resource administration that adjusts in actual time primarily based on load

• No guide configuration of GPU slices, MIG situations, or clusters

• Environment friendly GPU utilization with out overprovisioning or useful resource waste

• A constant and remoted runtime surroundings for all fashions

• Builders can give attention to purposes whereas Clarifai handles infrastructure

Compute Orchestration abstracts away the infrastructure work required to share GPUs successfully. You get higher utilization, smoother scaling, and 0 friction transferring from prototype to manufacturing. If you wish to discover additional, take a look at the getting began information.

Conclusion

On this weblog, we went over what GPU fractioning is and the way it works utilizing methods like TimeSlicing and MIG. These strategies allow you to run a number of fashions on the identical GPU by dividing up compute and reminiscence.

We additionally realized how Clarifai Compute Orchestration handles GPU fractioning on the orchestration layer. You possibly can spin up devoted compute tailor-made to your workloads, and Clarifai takes care of scheduling and scaling primarily based on demand.

Able to get began? Join Compute Orchestration at present and be a part of our Discord channel to attach with consultants and optimize your AI infrastructure!