mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-06-22 23:58:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
64a4351f3aba05bb220babc345fe006159d2ee24
LocalAI/docs/content/whats-new.md
LocalAI [bot] 600dafd20b feat(ced): sound-event classification backend (CED audio tagger) (#10425) 
* feat(ced): sketch sound-classification backend (CED audio tagger)

Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry,
footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend.

SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist
in DESIGN.md):
- backend/backend.proto: new SoundDetection rpc + SoundClass messages
  (run `make protogen-go` to regenerate pkg/grpc/proto).
- backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h),
  goced.go (Ced gRPC backend: Load + SoundDetection), Makefile
  (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh,
  package.sh, .gitignore.
- DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability
  registration checklist), gallery/index + CI registration, and a scoping
  note for the realtime/websocket live-recognition path (sliding-window
  classify over the existing ws transport + voicegate; the ced C-API
  per-PCM entry point is already window-friendly).

Backend code does not compile until protogen-go regenerates the pb types
and a libced.so is built (Makefile clones+builds it).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): REST /v1/audio/classification endpoint + capability registration

Wires the ced sound-event classification backend (AudioSet audio tagger)
end to end through the REST surface, mirroring the transcription path.

- Handler: core/http/endpoints/openai/sound_classification.go parses the
  multipart audio upload, temp-files it, resolves the model config and
  calls the SoundDetection RPC; returns {model, detections[]} JSON.
- Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection)
  loads the model and normalizes the proto response into schema types.
- Schema: core/schema/sound_classification.go (SoundClassificationResult).
- gRPC layer: SoundDetection wired through the LocalAI wrapper (interface,
  Backend client, Client, embed, server, base default) so the loader-typed
  client exposes the RPC; proto regenerated via make protogen-go.
- Route: POST /v1/audio/classification (+ /audio/classification alias) with
  the audio/multipart default-model middleware in routes/openai.go.
- Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_
  CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap +
  GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase
  option; /api/instructions audio area updated; auth RouteFeatureRegistry +
  FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI
  usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter
  + i18n; docs page features/audio-classification.md + whats-new + crosslink.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): realtime sound-event detection over the websocket API

When a realtime pipeline configures a sound-classification model, each
VAD-committed utterance (the same window the transcription path produces)
is also run through the CED sound-event classifier and the scored AudioSet
tags are emitted as a new server event. No new backend rpc is needed: the
SoundDetection gRPC method already exists on this branch.

- config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty)
  beside Transcription/VAD.
- realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the
  ModelInterface; implement it on wrappedModel and transcriptOnlyModel by
  calling backend.ModelSoundDetection with the session's sound-classification
  model config (mirrors how Transcribe dispatches). Load the optional config
  in newModel / newTranscriptionOnlyModel; nil config keeps it additive.
- types: add ConversationItemSoundDetectionEvent (item_id, content_index,
  detections[]{label,score,index}) with type conversation.item.sound_detection,
  its ServerEventType constant and MarshalJSON, mirroring the transcription
  completed event.
- realtime: add emitSoundDetection (unary path: classify the committed window,
  build the event, t.SendEvent) and wire it at the utterance-commit hook right
  after emitTranscription; gated on session.SoundDetectionEnabled (resolved
  from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0).
  Its error is logged via xlog but never aborts the turn.
- test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections,
  classifier error) plus a SoundDetection method on the fakeModel double.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ced): implement SoundDetection in nodes backend test doubles

The SoundDetection method added to the grpc backend interface left two
test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so
core/services/nodes failed to compile under `go vet`/`go test` (go build
missed it: the doubles live in _test.go). Add the method to both,
mirroring their existing Detect mock. Repairs CI for the nodes package.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): decouple realtime sound detection from VAD (sound-only sessions)

Sound-event detection must activate on sounds, not speech, so it no longer
runs through the voice VAD/transcription path. A sound-detection-only
pipeline (sound_detection set, no transcription/LLM) now:

- is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline
  stage),
- builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS
  loaded), and
- defaults the session to turn_detection none (no VAD) with no transcription
  stage, so the client drives windowing via input_audio_buffer.commit
  (option A: client-side sliding window). The per-PCM C-API already supports
  arbitrary windows.

commitUtterance gains a sound-only branch: it emits the
conversation.item.sound_detection event (scored AudioSet tags) and stops -
no transcription, no LLM response. generateResponse is now guarded on a
transcription stage being present, so a sound-only turn never invokes the LLM.

Existing transcription/VAD sessions are unchanged (additive). Added a
commitUtterance sound-only Ginkgo spec asserting it emits the sound event and
neither transcribes nor generates a response. go vet + golangci-lint
(new-from-merge-base) clean; openai suite green.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): register sound-classification backend in gallery + CI

Mechanical backend-image registration for the ced sound-event classifier,
mirroring the parakeet-cpp Go/purego backend everywhere it is wired up.

- .github/backend-matrix.yml: add the ced build matrix, field-for-field copies
  of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64,
  l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan
  amd64/arm64, rocm hipblas, and the metal darwin entry), changing only
  backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang.
- backend/index.yaml: add the &ced meta anchor (capabilities map per platform)
  plus ced-development and the per-arch image entries, each uri/mirror
  tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is
  intentionally deferred pending the HuggingFace publish (TODO note inline).
- scripts/changed-backends.js: add an explicit item.backend === "ced" branch in
  inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as
  the parakeet-cpp branch (before the generic golang fallthrough).
- .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in
  backend/go/ced/Makefile so the daily bot bumps the pin.
- swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so
  the existing /v1/audio/classification annotations land in the generated spec.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): server-side windowing for realtime sound detection (option B)

Adds an optional server-driven sliding-window classifier so a sound-only
realtime client only has to stream audio (no input_audio_buffer.commit):

- Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs.
  When both > 0 on a sound-only session, the server classifies the last
  window of streamed audio every hop and emits a conversation.item.sound_
  detection event; the input buffer is trimmed to one window so a long
  stream stays bounded. When unset, the session stays client-driven
  (option A). Runs independent of VAD (sound events are not speech).
- handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so
  it is unit-testable) + writeWindowWAV, which declares the true
  InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples
  correctly. Goroutine is started after toggleVAD and torn down with the
  session (close + wg.Wait).
- Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta
  registry; the earlier realtime commit added pipeline.sound_detection
  without a registry entry, failing TestAllFieldsHaveRegistryEntries. This
  fixes that and covers the two new knobs.

Tests: classifySoundWindow emits an event + trims the buffer to one window,
no-ops on too-little audio; writeWindowWAV declares the given sample rate.
go build/vet + golangci-lint (new-from-merge-base) clean; config + openai
suites green.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0)

The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0,
converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced +
known_usecases: sound_classification) and two gallery/index.yaml entries
(ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and
removes the now-resolved TODO from backend/index.yaml.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): add tiny/mini/small GGUF model gallery entries

Publishes the rest of the CED family (same architecture, metadata-driven port
verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds
their f16 + q8_0 gallery entries:

  ced-tiny  (5.5M, edge/Pi-class)  f16 11MB / q8_0 6MB
  ced-mini  (9.6M)                 f16 19MB / q8_0 11MB
  ced-small (22M)                  f16 42MB / q8_0 23MB

All sha256-pinned. ced-base remains the accuracy default.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo

All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single
HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8
gallery model entries' urls + file uris accordingly. sha256 and filenames are
unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): bump CED_VERSION to the short-clip fix

Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip
shorter than target_length (~10.11s): time_pos_embed was added at its full
63-frame grid instead of being sliced to the clip's actual time grid, tripping
ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s
windows) and gated with a short-clip parity test upstream.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive

- README.md: add ced.cpp to the "native C/C++/GGML engines developed and
  maintained by the LocalAI project" table.
- docs/content/features/backends.md: add a Sound Classification backend
  category (sound-event classification / audio tagging) listing ced.cpp.
- .agents/adding-backends.md: add a "Documenting the backend" section and two
  verification-checklist items requiring new backends to be documented in the
  backends.md category list, and in-house native engines to be added to the
  README maintained-engines table. This directive was missing.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): repin CED_VERSION to the v0.1.0 release commit

ced.cpp history was squashed into a single release commit (tagged v0.1.0), so
the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the
v0.1.0 release commit, so the backend builds against a commit that exists.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths

- sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of
  the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler.
- goced.go: reading a NUL-terminated C string from a libced-owned buffer.
  #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since
  the uintptr is a C-owned malloc'd buffer, not Go-GC memory.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-06-22 01:00:28 +02:00

30 KiB
Raw Blame History

+++ disableToc = false title = "News" weight = 7 url = '/basics/news/' icon = "newspaper" +++

Release notes have been now moved completely over Github releases.

You can see the release notes here.

2026 Highlights

  • April 2026: Audio Transform — generic audio-in / audio-out endpoint with optional reference signal. First implementation: LocalVQE C++ backend (joint AEC + noise suppression + dereverberation, DeepVQE-style). Both batch (POST /audio/transformations) and bidirectional WebSocket streaming (/audio/transformations/stream). Studio "Transform" tab with synchronized waveform players for input / reference / output.
  • April 2026: Face recognition backend — insightface-powered 1:1 verification, 1:N identification, face embedding, face detection, and demographic analysis. Ships both a non-commercial buffalo_l model and an Apache 2.0 OpenCV Zoo alternative.
  • May 2026: Speaker diarization — new /v1/audio/diarization endpoint returning "who spoke when" segments. Backed by sherpa-onnx (pyannote-3.0 + speaker embeddings + clustering) for pure diarization, and vibevoice-cpp for diarization bundled with long-form ASR. Supports json / verbose_json / rttm response formats.
  • June 2026: Sound classification — new /v1/audio/classification endpoint for audio tagging / sound-event classification, returning scored AudioSet labels (baby cry, glass breaking, alarms, ...). Backed by ced.cpp, a 527-class AudioSet tagger ported to ggml.
  • June 2026: PII analyze / redact API — the PII detection pipeline (NER + restricted-regex pattern tiers) is now a standalone service: POST /api/pii/analyze returns detected entity spans and POST /api/pii/redact returns the sanitised text (or 400 pii_blocked), without routing a chat request through the middleware. Events gain an origin (middleware / proxy / pii_analyze / pii_redact) so /api/pii/events can be filtered by source.
  • June 2026: Concurrent scoring and PII NER on llama.cpp — the Score (router classifier) and TokenClassify (PII NER) primitives now ride llama.cpp's server task queue instead of locking the context, so they run concurrently with chat/completion/embedding traffic and with each other. The known_usecases restriction that forced dedicated scorer/NER model configs on llama-cpp is lifted, repeated scoring calls reuse the prompt KV cache across candidates, and scoring inputs are no longer capped by the physical batch size.

2024 Highlights

04-12-2023: v2.0.0

This release brings a major overhaul in some backends.

Breaking/important changes:

  • Backend rename: llama-stable renamed to llama-ggml {{< pr "1287" >}}
  • Prompt template changes: {{< pr "1254" >}} (extra space in roles)
  • Apple metal bugfixes: {{< pr "1365" >}}

New:

  • Added support for LLaVa and OpenAI Vision API support ({{< pr "1254" >}})
  • Python based backends are now using conda to track env dependencies ( {{< pr "1144" >}} )
  • Support for parallel requests ( {{< pr "1290" >}} )
  • Support for transformers-embeddings ( {{< pr "1308" >}})
  • Watchdog for backends ( {{< pr "1341" >}}). As https://github.com/ggerganov/llama.cpp/issues/3969 is hitting LocalAI's llama-cpp implementation, we have now a watchdog that can be used to make sure backends are not stalling. This is a generic mechanism that can be enabled for all the backends now.
  • Whisper.cpp updates ( {{< pr "1302" >}} )
  • Petals backend ( {{< pr "1350" >}} )
  • Full LLM fine-tuning example to use with LocalAI: https://localai.io/advanced/fine-tuning/

Due to the python dependencies size of images grew in size. If you still want to use smaller images without python dependencies, you can use the corresponding images tags ending with -core.

Full changelog: https://github.com/mudler/LocalAI/releases/tag/v2.0.0

30-10-2023: v1.40.0

This release is a preparation before v2 - the efforts now will be to refactor, polish and add new backends. Follow up on: https://github.com/mudler/LocalAI/issues/1126

Hot topics

This release now brings the llama-cpp backend which is a c++ backend tied to llama.cpp. It follows more closely and tracks recent versions of llama.cpp. It is not feature compatible with the current llama backend but plans are to sunset the current llama backend in favor of this one. This one will be probably be the latest release containing the older llama backend written in go and c++. The major improvement with this change is that there are less layers that could be expose to potential bugs - and as well it ease out maintenance as well.

Support for ROCm/HIPBLAS

This release bring support for AMD thanks to @65a . See more details in {{< pr "1100" >}}

More CLI commands

Thanks to @jespino now the local-ai binary has more subcommands allowing to manage the gallery or try out directly inferencing, check it out!

Release notes

25-09-2023: v1.30.0

This is an exciting LocalAI release! Besides bug-fixes and enhancements this release brings the new backend to a whole new level by extending support to vllm and vall-e-x for audio generation!

Check out the documentation for vllm here and Vall-E-X here

Release notes

26-08-2023: v1.25.0

Hey everyone, Ettore here, I'm so happy to share this release out - while this summer is hot apparently doesn't stop LocalAI development :)

This release brings a lot of new features, bugfixes and updates! Also a big shout out to the community, this was a great release!

Attention 🚨

From this release the llama backend supports only gguf files (see {{< pr "943" >}}). LocalAI however still supports ggml files. We ship a version of llama.cpp before that change in a separate backend, named llama-stable to allow still loading ggml files. If you were specifying the llama backend manually to load ggml files from this release you should use llama-stable instead, or do not specify a backend at all (LocalAI will automatically handle this).

Image generation enhancements

The [Diffusers]({{%relref "features/image-generation" %}}) backend got now various enhancements, including support to generate images from images, longer prompts, and support for more kernels schedulers. See the [Diffusers]({{%relref "features/image-generation" %}}) documentation for more information.

Lora adapters

Now it's possible to load lora adapters for llama.cpp. See {{< pr "955" >}} for more information.

Device management

It is now possible for single-devices with one GPU to specify --single-active-backend to allow only one backend active at the time {{< pr "925" >}}.

Community spotlight

Resources management

Thanks to the continuous community efforts (another cool contribution from {{< github "dave-gray101" >}} ) now it's possible to shutdown a backend programmatically via the API. There is an ongoing effort in the community to better handling of resources. See also the 🔥Roadmap.

New how-to section

Thanks to the community efforts now we have a new how-to website with various examples on how to use LocalAI. This is a great starting point for new users! We are currently working on improving it, a huge shout out to {{< github "lunamidori5" >}} from the community for the impressive efforts on this!

💡 More examples!

  • Open source autopilot? See the new addition by {{< github "gruberdev" >}} in our examples on how to use Continue with LocalAI!
  • Want to try LocalAI with Insomnia? Check out the new Insomnia example by {{< github "dave-gray101" >}}!

LocalAGI in discord!

Did you know that we have now few cool bots in our Discord? come check them out! We also have an instance of LocalAGI ready to help you out!

Changelog summary

Breaking Changes 🛠

  • feat: bump llama.cpp, add gguf support by {{< github "mudler" >}} in {{< pr "943" >}}

Exciting New Features 🎉

  • feat(Makefile): allow to restrict backend builds by {{< github "mudler" >}} in {{< pr "890" >}}
  • feat(diffusers): various enhancements by {{< github "mudler" >}} in {{< pr "895" >}}
  • feat: make initializer accept gRPC delay times by {{< github "mudler" >}} in {{< pr "900" >}}
  • feat(diffusers): add DPMSolverMultistepScheduler++, DPMSolverMultistepSchedulerSDE++, guidance_scale by {{< github "mudler" >}} in {{< pr "903" >}}
  • feat(diffusers): overcome prompt limit by {{< github "mudler" >}} in {{< pr "904" >}}
  • feat(diffusers): add img2img and clip_skip, support more kernels schedulers by {{< github "mudler" >}} in {{< pr "906" >}}
  • Usage Features by {{< github "dave-gray101" >}} in {{< pr "863" >}}
  • feat(diffusers): be consistent with pipelines, support also depthimg2img by {{< github "mudler" >}} in {{< pr "926" >}}
  • feat: add --single-active-backend to allow only one backend active at the time by {{< github "mudler" >}} in {{< pr "925" >}}
  • feat: add llama-stable backend by {{< github "mudler" >}} in {{< pr "932" >}}
  • feat: allow to customize rwkv tokenizer by {{< github "dave-gray101" >}} in {{< pr "937" >}}
  • feat: backend monitor shutdown endpoint, process based by {{< github "dave-gray101" >}} in {{< pr "938" >}}
  • feat: Allow to load lora adapters for llama.cpp by {{< github "mudler" >}} in {{< pr "955" >}}

Join our Discord community! our vibrant community is growing fast, and we are always happy to help! https://discord.gg/uJAeKSAGDy

The full changelog is available here.

🔥🔥🔥🔥 12-08-2023: v1.24.0 🔥🔥🔥🔥

This is release brings four(!) new additional backends to LocalAI: [🐶 Bark]({{%relref "features/text-to-audio#bark" %}}), 🦙 [AutoGPTQ]({{%relref "features/text-generation#autogptq" %}}), [🧨 Diffusers]({{%relref "features/image-generation" %}}), 🦙 [exllama]({{%relref "features/text-generation#exllama" %}}) and a lot of improvements!

Major improvements:

  • feat: add bark and AutoGPTQ by {{< github "mudler" >}} in {{< pr "871" >}}
  • feat: Add Diffusers by {{< github "mudler" >}} in {{< pr "874" >}}
  • feat: add API_KEY list support by {{< github "neboman11" >}} and {{< github "bnusunny" >}} in {{< pr "877" >}}
  • feat: Add exllama by {{< github "mudler" >}} in {{< pr "881" >}}
  • feat: pre-configure LocalAI galleries by {{< github "mudler" >}} in {{< pr "886" >}}

🐶 Bark

[Bark]({{%relref "features/text-to-audio#bark" %}}) is a text-prompted generative audio model - it combines GPT techniques to generate Audio from text. It is a great addition to LocalAI, and it's available in the container images by default.

It can also generate music, see the example: lion.webm

🦙 AutoGPTQ

[AutoGPTQ]({{%relref "features/text-generation#autogptq" %}}) is an easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

It is targeted mainly for GPU usage only. Check out the [ documentation]({{%relref "features/text-generation" %}}) for usage.

🦙 Exllama

[Exllama]({{%relref "features/text-generation#exllama" %}}) is a "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights". It is a faster alternative to run LLaMA models on GPU.Check out the [Exllama documentation]({{%relref "features/text-generation#exllama" %}}) for usage.

🧨 Diffusers

[Diffusers]({{%relref "features/image-generation#diffusers" %}}) is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Currently it is experimental, and supports generation only of images so you might encounter some issues on models which weren't tested yet. Check out the [Diffusers documentation]({{%relref "features/image-generation" %}}) for usage.

🔑 API Keys

Thanks to the community contributions now it's possible to specify a list of API keys that can be used to gate API requests.

API Keys can be specified with the API_KEY environment variable as a comma-separated list of keys.

🖼️ Galleries

Now by default the model-gallery repositories are configured in the container images

💡 New project

LocalAGI is a simple agent that uses LocalAI functions to have a full locally runnable assistant (with no API keys needed).

See it here in action planning a trip for San Francisco!

The full changelog is available here.

🔥🔥 29-07-2023: v1.23.0 🚀

This release focuses mostly on bugfixing and updates, with just a couple of new features:

  • feat: add rope settings and negative prompt, drop grammar backend by {{< github "mudler" >}} in {{< pr "797" >}}
  • Added CPU information to entrypoint.sh by @finger42 in {{< pr "794" >}}
  • feat: cancel stream generation if client disappears by @tmm1 in {{< pr "792" >}}

Most notably, this release brings important fixes for CUDA (and not only):

  • fix: add rope settings during model load, fix CUDA by {{< github "mudler" >}} in {{< pr "821" >}}
  • fix: select function calls if 'name' is set in the request by {{< github "mudler" >}} in {{< pr "827" >}}
  • fix: symlink libphonemize in the container by {{< github "mudler" >}} in {{< pr "831" >}}

{{% notice note %}}

From this release [OpenAI functions]({{%relref "features/openai-functions" %}}) are available in the llama backend. The llama-grammar has been deprecated. See also [OpenAI functions]({{%relref "features/openai-functions" %}}).

{{% /notice %}}

The full changelog is available here

🔥🔥🔥 23-07-2023: v1.22.0 🚀

  • feat: add llama-master backend by {{< github "mudler" >}} in {{< pr "752" >}}
  • [build] pass build type to cmake on libtransformers.a build by @TonDar0n in {{< pr "741" >}}
  • feat: resolve JSONSchema refs (planners) by {{< github "mudler" >}} in {{< pr "774" >}}
  • feat: backends improvements by {{< github "mudler" >}} in {{< pr "778" >}}
  • feat(llama2): add template for chat messages by {{< github "dave-gray101" >}} in {{< pr "782" >}}

{{% notice note %}}

From this release to use the OpenAI functions you need to use the llama-grammar backend. It has been added a llama backend for tracking llama.cpp master and llama-grammar for the grammar functionalities that have not been merged yet upstream. See also [OpenAI functions]({{%relref "features/openai-functions" %}}). Until the feature is merged we will have two llama backends.

{{% /notice %}}

Huggingface embeddings

In this release is now possible to specify to LocalAI external gRPC backends that can be used for inferencing {{< pr "778" >}}. It is now possible to write internal backends in any language, and a huggingface-embeddings backend is now available in the container image to be used with https://github.com/UKPLab/sentence-transformers. See also [Embeddings]({{%relref "features/embeddings" %}}).

LLaMa 2 has been released!

Thanks to the community effort now LocalAI supports templating for LLaMa2! more at: {{< pr "782" >}} until we update the model gallery with LLaMa2 models!

Official langchain integration

Progress has been made to support LocalAI with langchain. See: https://github.com/langchain-ai/langchain/pull/8134

🔥🔥🔥 17-07-2023: v1.21.0 🚀

  • [whisper] Partial support for verbose_json format in transcribe endpoint by @ldotlopez in {{< pr "721" >}}
  • LocalAI functions by @mudler in {{< pr "726" >}}
  • gRPC-based backends by @mudler in {{< pr "743" >}}
  • falcon support (7b and 40b) with ggllm.cpp by @mudler in {{< pr "743" >}}

LocalAI functions

This allows to run OpenAI functions as described in the OpenAI blog post and documentation: https://openai.com/blog/function-calling-and-other-api-updates.

This is a video of running the same example, locally with LocalAI: localai-functions-1

And here when it actually picks to reply to the user instead of using functions! functions-2

Note: functions are supported only with llama.cpp-compatible models.

A full example is available here: https://github.com/mudler/LocalAI-examples/tree/main/functions

gRPC backends

This is an internal refactor which is not user-facing, however, it allows to ease out maintenance and addition of new backends to LocalAI!

falcon support

Now Falcon 7b and 40b models compatible with https://github.com/cmp-nct/ggllm.cpp are supported as well.

The former, ggml-based backend has been renamed to falcon-ggml.

Default pre-compiled binaries

From this release the default behavior of images has changed. Compilation is not triggered on start automatically, to recompile local-ai from scratch on start and switch back to the old behavior, you can set REBUILD=true in the environment variables. Rebuilding can be necessary if your CPU and/or architecture is old and the pre-compiled binaries are not compatible with your platform. See the [build section]({{%relref "installation/build" %}}) for more information.

Full release changelog

🔥🔥🔥 28-06-2023: v1.20.0 🚀

Exciting New Features 🎉

  • Add Text-to-Audio generation with go-piper by {{< github "mudler" >}} in {{< pr "649" >}} See [API endpoints]({{%relref "features/text-to-audio" %}}) in our documentation.
  • Add gallery repository by {{< github "mudler" >}} in {{< pr "663" >}}. See [models]({{%relref "features/model-gallery" %}}) for documentation.

Container images

  • Standard (GPT + stablediffusion): quay.io/go-skynet/local-ai:v1.20.0
  • FFmpeg: quay.io/go-skynet/local-ai:v1.20.0-ffmpeg
  • CUDA 11+FFmpeg: quay.io/go-skynet/local-ai:v1.20.0-gpu-nvidia-cuda11-ffmpeg
  • CUDA 12+FFmpeg: quay.io/go-skynet/local-ai:v1.20.0-gpu-nvidia-cuda12-ffmpeg

Updates

Updates to llama.cpp, go-transformers, gpt4all.cpp and rwkv.cpp.

The NUMA option was enabled by {{< github "mudler" >}} in {{< pr "684" >}}, along with many new parameters (mmap,mmlock, ..). See [advanced]({{%relref "advanced" %}}) for the full list of parameters.

Gallery repositories

In this release there is support for gallery repositories. These are repositories that contain models, and can be used to install models. The default gallery which contains only freely licensed models is in Github: https://github.com/go-skynet/model-gallery, but you can use your own gallery by setting the GALLERIES environment variable. An automatic index of huggingface models is available as well.

For example, now you can start LocalAI with the following environment variable to use both galleries:

GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:ci-robbot/localai-huggingface-zoo/index.yaml","name":"huggingface"}]

And in runtime you can install a model from huggingface now with:

curl http://localhost:8000/models/apply -H "Content-Type: application/json" -d '{ "id": "huggingface@thebloke__open-llama-7b-open-instruct-ggml__open-llama-7b-open-instruct.ggmlv3.q4_0.bin" }'

or a tts voice with:

curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{ "id": "model-gallery@voice-en-us-kathleen-low" }'

See also [models]({{%relref "features/model-gallery" %}}) for a complete documentation.

Text to Audio

Now LocalAI uses piper and go-piper to generate audio from text. This is an experimental feature, and it requires GO_TAGS=tts to be set during build. It is enabled by default in the pre-built container images.

To setup audio models, you can use the new galleries, or setup the models manually as described in [the API section of the documentation]({{%relref "features/text-to-audio" %}}).

You can check the full changelog in Github

🔥🔥🔥 19-06-2023: v1.19.0 🚀

  • Full CUDA GPU offload support ( PR by mudler. Thanks to chnyda for handing over the GPU access, and lu-zero to help in debugging )
  • Full GPU Metal Support is now fully functional. Thanks to Soleblaze to iron out the Metal Apple silicon support!

Container images:

  • Standard (GPT + stablediffusion): quay.io/go-skynet/local-ai:v1.19.2
  • FFmpeg: quay.io/go-skynet/local-ai:v1.19.2-ffmpeg
  • CUDA 11+FFmpeg: quay.io/go-skynet/local-ai:v1.19.2-gpu-nvidia-cuda11-ffmpeg
  • CUDA 12+FFmpeg: quay.io/go-skynet/local-ai:v1.19.2-gpu-nvidia-cuda12-ffmpeg

🔥🔥🔥 06-06-2023: v1.18.0 🚀

This LocalAI release is plenty of new features, bugfixes and updates! Thanks to the community for the help, this was a great community release!

We now support a vast variety of models, while being backward compatible with prior quantization formats, this new release allows still to load older formats and new k-quants!

New features

  • ✨ Added support for falcon-based model families (7b) ( mudler )
  • ✨ Experimental support for Metal Apple Silicon GPU - ( mudler and thanks to Soleblaze for testing! ). See the [build section]({{%relref "installation/build#Acceleration" %}}).
  • ✨ Support for token stream in the /v1/completions endpoint ( samm81 )
  • ✨ Added huggingface backend ( Evilfreelancer )
  • 📷 Stablediffusion now can output 2048x2048 images size with esrgan! ( mudler )

Container images

Dependencies updates

  • 🆙 Bloomz has been updated to the latest ggml changes, including new quantization format ( mudler )
  • 🆙 RWKV has been updated to the new quantization format( mudler )
  • 🆙 k-quants format support for the llama models ( mudler )
  • 🆙 gpt4all has been updated, incorporating upstream changes allowing to load older models, and with different CPU instruction set (AVX only, AVX2) from the same binary! ( mudler )

Generic

  • 🐧 Fully Linux static binary releases ( mudler )
  • 📷 Stablediffusion has been enabled on container images by default ( mudler ) Note: You can disable container image rebuilds with REBUILD=false

Examples

Two new projects offer now direct integration with LocalAI!

Full release changelog

29-05-2023: v1.17.0

Support for OpenCL has been added while building from sources.

You can now build LocalAI from source with BUILD_TYPE=clblas to have an OpenCL build. See also the [build section]({{%relref "getting-started/build#Acceleration" %}}).

For instructions on how to install OpenCL/CLBlast see here.

rwkv.cpp has been updated to the new ggml format commit.

27-05-2023: v1.16.0

Now it's possible to automatically download pre-configured models before starting the API.

Start local-ai with the PRELOAD_MODELS containing a list of models from the gallery, for instance to install gpt4all-j as gpt-3.5-turbo:

PRELOAD_MODELS=[{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo"}]

llama.cpp models now can also automatically save the prompt cache state as well by specifying in the model YAML configuration file:


prompt_cache_path: "alpaca-cache"

prompt_cache_all: true

See also the [advanced section]({{%relref "advanced" %}}).

Media, Blogs, Social

Previous

  • 23-05-2023: v1.15.0 released. go-gpt2.cpp backend got renamed to go-ggml-transformers.cpp updated including https://github.com/ggerganov/llama.cpp/pull/1508 which breaks compatibility with older models. This impacts RedPajama, GptNeoX, MPT(not gpt4all-mpt), Dolly, GPT2 and Starcoder based models. Binary releases available, various fixes, including {{< pr "341" >}} .
  • 21-05-2023: v1.14.0 released. Minor updates to the /models/apply endpoint, llama.cpp backend updated including https://github.com/ggerganov/llama.cpp/pull/1508 which breaks compatibility with older models. gpt4all is still compatible with the old format.
  • 19-05-2023: v1.13.0 released! 🔥🔥 updates to the gpt4all and llama backend, consolidated CUDA support ( {{< pr "310" >}} thanks to @bubthegreat and @Thireus ), preliminar support for [installing models via API]({{%relref "advanced#" %}}).
  • 17-05-2023: v1.12.0 released! 🔥🔥 Minor fixes, plus CUDA ({{< pr "258" >}}) support for llama.cpp-compatible models and image generation ({{< pr "272" >}}).
  • 16-05-2023: 🔥🔥🔥 Experimental support for CUDA ({{< pr "258" >}}) in the llama.cpp backend and Stable diffusion CPU image generation ({{< pr "272" >}}) in master.

Now LocalAI can generate images too:

mode=0 mode=1 (winograd/sgemm)
b6441997879 winograd2
  • 14-05-2023: v1.11.1 released! rwkv backend patch release
  • 13-05-2023: v1.11.0 released! 🔥 Updated llama.cpp bindings: This update includes a breaking change in the model files ( https://github.com/ggerganov/llama.cpp/pull/1405 ) - old models should still work with the gpt4all-llama backend.
  • 12-05-2023: v1.10.0 released! 🔥🔥 Updated gpt4all bindings. Added support for GPTNeox (experimental), RedPajama (experimental), Starcoder (experimental), Replit (experimental), MosaicML MPT. Also now embeddings endpoint supports tokens arrays. See the langchain-chroma example! Note - this update does NOT include https://github.com/ggerganov/llama.cpp/pull/1405 which makes models incompatible.
  • 11-05-2023: v1.9.0 released! 🔥 Important whisper updates ( {{< pr "233" >}} {{< pr "229" >}} ) and extended gpt4all model families support ( {{< pr "232" >}} ). Redpajama/dolly experimental ( {{< pr "214" >}} )
  • 10-05-2023: v1.8.0 released! 🔥 Added support for fast and accurate embeddings with bert.cpp ( {{< pr "222" >}} )
  • 09-05-2023: Added experimental support for transcriptions endpoint ( {{< pr "211" >}} )
  • 08-05-2023: Support for embeddings with models using the llama.cpp backend ( {{< pr "207" >}} )
  • 02-05-2023: Support for rwkv.cpp models ( {{< pr "158" >}} ) and for /edits endpoint
  • 01-05-2023: Support for SSE stream of tokens in llama.cpp backends ( {{< pr "152" >}} )
Reference in New Issue View Git Blame Copy Permalink