Skip to main content
Every time a new runner starts, it needs Docker image layers, model weights, and potentially compiled model caches before it can serve requests. Without caching, each runner would download everything from scratch, leading to cold starts that can take minutes for large models. fal’s caching system stores these artifacts at multiple levels so that subsequent runners start faster, and cold start times improve progressively as your app serves traffic. Docker layers and files on /data are cached automatically. You don’t need to configure anything. Compiled model caches (PyTorch Inductor) require explicit use of the synchronized_inductor_cache() API. Understanding the cache layers helps you reason about why cold starts improve over time and how to structure your app for the best startup performance. For strategies to reduce cold starts further, see Optimizing Cold Starts.

​
Cache Layers

Docker images and /data storage use different caching mechanisms.

​
/data Storage Cache

Files on /data (model weights, compiled caches, any files your app writes) go through a three-layer cache: When a runner reads a file from /data, it checks the local node cache first. If the file isn’t found locally, it checks the distributed datacenter cache. If that also misses, it fetches from the object store and populates both caches on the way back. This means the first runner in a region pays the full download cost, but every runner after it benefits from progressively faster access.

​
Docker Image Cache

Each node keeps Docker images in a local image store, separate from the /data cache. When a runner starts, the scheduler prefers nodes that already have your image. This preference is not a guarantee. The scheduler also considers GPU availability and region, so a runner can start on a node that must pull the image first. When the node does not have the image, it pulls the image from the container registry. After you deploy a new revision, fal can pull its image in advance onto nodes in the datacenters where your app runs. fal prefers nodes that have the image of the previous revision, because these nodes already have most of the layers. When a node needs disk space, it removes the least recently used images first. If your app does not run for some time, a node can pull its image again. fal keeps a BuildKit registry cache for each app. When you change a small part of your Dockerfile, fal rebuilds only the affected layers. When you deploy without changes to the image, fal uses the existing image and does not build again. To force a rebuild, pass --no-cache to fal deploy. The build cache is separate for each environment. For example, you can deploy a commit to a staging environment and then to the main environment. The main build does not use the layers from the staging build.

​
What Gets Cached

Docker Image Layers: Docker images are split into layers, and each layer is cached independently. If you change a single dependency in your Dockerfile, fal rebuilds only the affected layers. A node that has the previous image pulls only the changed layers. See Optimize Container Images for tips on structuring your Dockerfile for better layer caching. Model Weights: files downloaded to /data are automatically cached across runners through the three-layer /data cache described above. This includes Hugging Face models (cached at /data/.cache/huggingface via the HF_HOME environment variable), weights downloaded with download_model_weights(), and any other files you write to /data. The first runner downloads the weights; subsequent runners read from the local or distributed cache. Compiled Model Caches: PyTorch Inductor compiled models and other JIT compilation artifacts. If you use torch.compile(), you can persist and share the resulting compiled kernels across runners using synchronized_inductor_cache(). This stores the cache on /data (at /data/inductor-caches/<GPU_TYPE>/), where it benefits from the same three-layer cache. See Compiled Kernel Caching for setup instructions.

​
How Caches Warm Up

Caches warm progressively as your application serves traffic:
  1. First runner: pulls the Docker image from the registry, unless fal pulled it in advance after the deploy. Downloads model weights from the object store. Populates the local and distributed /data caches. This is the slowest cold start.
  2. Second runner on the same node: the Docker image is already in the node’s image store, and model weights are in the local NVMe cache. Cold start is significantly faster.
  3. Runners on other nodes: model weights are served from the distributed /data cache. The scheduler prefers nodes that already have the Docker image. If the runner starts on a node without the image, the node pulls it from the registry.
  4. Over time: as more nodes serve your app, caches spread across the region. Cold starts converge to the time it takes to run setup() with all files already available locally.
Caches are not permanent. Local caches can be evicted when nodes are recycled or need disk space. If your app hasn’t had traffic for a while, the first cold start after a quiet period can be slower as caches are repopulated.