mirror of
https://github.com/mudler/LocalAI.git
synced 2026-01-25 06:42:46 -05:00
6644af10c6
feat: reduce images size and stop bundling sources Do not copy sources anymore, and reduce packages of the base images by not using builder images. If needed to rebuild, just build the container image from scratch by following the docs. We will slowly try to migrate all backends to the gallery to keep the core small. This PR is a breaking change, it also sets the base folders to /models and /backends instead of /build/models and /build/backends. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
1.5 KiB
1.5 KiB
+++ disableToc = false title = "Running on Nvidia ARM64" weight = 27 +++
LocalAI can be run on Nvidia ARM64 devices, such as the Jetson Nano, Jetson Xavier NX, and Jetson AGX Xavier. The following instructions will guide you through building the LocalAI container for Nvidia ARM64 devices.
Prerequisites
- Docker engine installed (https://docs.docker.com/engine/install/ubuntu/)
- Nvidia container toolkit installed (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-ap)
Build the container
Build the LocalAI container for Nvidia ARM64 devices using the following command:
git clone https://github.com/mudler/LocalAI
cd LocalAI
docker build --build-arg SKIP_DRIVERS=true --build-arg BUILD_TYPE=cublas --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core .
Otherwise images are available on quay.io and dockerhub:
docker pull quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core
Usage
Run the LocalAI container on Nvidia ARM64 devices using the following command, where /data/models is the directory containing the models:
docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core
Note: /data/models is the directory containing the models. You can replace it with the directory containing your models.