mirror of
https://github.com/navidrome/navidrome.git
synced 2025-12-23 23:18:05 -05:00
feat(scanner): implement selective folder scanning and file system watcher improvements (#4674)
* feat: Add selective folder scanning capability Implement targeted scanning of specific library/folder pairs without full recursion. This enables efficient rescanning of individual folders when changes are detected, significantly reducing scan time for large libraries. Key changes: - Add ScanTarget struct and ScanFolders API to Scanner interface - Implement CLI flag --targets for specifying libraryID:folderPath pairs - Add FolderRepository.GetByPaths() for batch folder info retrieval - Create loadSpecificFolders() for non-recursive directory loading - Scope GC operations to affected libraries only (with TODO for full impl) - Add comprehensive tests for selective scanning behavior The selective scan: - Only processes specified folders (no subdirectory recursion) - Maintains library isolation - Runs full maintenance pipeline scoped to affected libraries - Supports both full and quick scan modes Examples: navidrome scan --targets "1:Music/Rock,1:Music/Jazz" navidrome scan --full --targets "2:Classical" * feat(folder): replace GetByPaths with GetFolderUpdateInfo for improved folder updates retrieval Signed-off-by: Deluan <deluan@navidrome.org> * test: update parseTargets test to handle folder names with spaces Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder): remove unused LibraryPath struct and update GC logging message Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder): enhance external scanner to support target-specific scanning Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify scanner methods Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): implement folder scanning notifications with deduplication Signed-off-by: Deluan <deluan@navidrome.org> * refactor(watcher): add resolveFolderPath function for testability Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): implement path ignoring based on .ndignore patterns Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): implement IgnoreChecker for managing .ndignore patterns Signed-off-by: Deluan <deluan@navidrome.org> * refactor(ignore_checker): rename scanner to lineScanner for clarity Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): enhance ScanTarget struct with String method for better target representation Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner): validate library ID to prevent negative values Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify GC method by removing library ID parameter Signed-off-by: Deluan <deluan@navidrome.org> * feat(scanner): update folder scanning to include all descendants of specified folders Signed-off-by: Deluan <deluan@navidrome.org> * feat(subsonic): allow selective scan in the /startScan endpoint Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): update CallScan to handle specific library/folder pairs Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): streamline scanning logic by removing scanAll method Signed-off-by: Deluan <deluan@navidrome.org> * test: enhance mockScanner for thread safety and improve test reliability Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): move scanner.ScanTarget to model.ScanTarget Signed-off-by: Deluan <deluan@navidrome.org> * refactor: move scanner types to model,implement MockScanner Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): update scanner interface and implementations to use model.Scanner Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder_repository): normalize target path handling by using filepath.Clean Signed-off-by: Deluan <deluan@navidrome.org> * test(folder_repository): add comprehensive tests for folder retrieval and child exclusion Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify selective scan logic using slice.Filter Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): streamline phase folder and album creation by removing unnecessary library parameter Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): move initialization logic from phase_1 to the scanner itself Signed-off-by: Deluan <deluan@navidrome.org> * refactor(tests): rename selective scan test file to scanner_selective_test.go Signed-off-by: Deluan <deluan@navidrome.org> * feat(configuration): add DevSelectiveWatcher configuration option Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): enhance .ndignore handling for folder deletions and file changes Signed-off-by: Deluan <deluan@navidrome.org> * docs(scanner): comments Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): enhance walkDirTree to support target folder scanning Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner, watcher): handle errors when pushing ignore patterns for folders Signed-off-by: Deluan <deluan@navidrome.org> * Update scanner/phase_1_folders.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor(scanner): replace parseTargets function with direct call to scanner.ParseTargets Signed-off-by: Deluan <deluan@navidrome.org> * test(scanner): add tests for ScanBegin and ScanEnd functionality Signed-off-by: Deluan <deluan@navidrome.org> * fix(library): update PRAGMA optimize to check table sizes without ANALYZE Signed-off-by: Deluan <deluan@navidrome.org> * test(scanner): refactor tests Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add selective scan options and update translations Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add quick and full scan options for individual libraries Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add Scan buttonsto the LibraryList Signed-off-by: Deluan <deluan@navidrome.org> * feat(scan): update scanning parameters from 'path' to 'target' for selective scans. * refactor(scan): move ParseTargets function to model package * test(scan): suppress unused return value from SetUserLibraries in tests * feat(gc): enhance garbage collection to support selective library purging Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner): prevent race condition when scanning deleted folders When the watcher detects changes in a folder that gets deleted before the scanner runs (due to the 10-second delay), the scanner was prematurely removing these folders from the tracking map, preventing them from being marked as missing. The issue occurred because `newFolderEntry` was calling `popLastUpdate` before verifying the folder actually exists on the filesystem. Changes: - Move fs.Stat check before newFolderEntry creation in loadDir to ensure deleted folders remain in lastUpdates for finalize() to handle - Add early existence check in walkDirTree to skip non-existent target folders with a warning log - Add unit test verifying non-existent folders aren't removed from lastUpdates prematurely - Add integration test for deleted folder scenario with ScanFolders Fixes the issue where deleting entire folders (e.g., /music/AC_DC) wouldn't mark tracks as missing when using selective folder scanning. * refactor(scan): streamline folder entry creation and update handling Signed-off-by: Deluan <deluan@navidrome.org> * feat(scan): add '@Recycle' (QNAP) to ignored directories list Signed-off-by: Deluan <deluan@navidrome.org> * fix(log): improve thread safety in logging level management * test(scan): move unit tests for ParseTargets function Signed-off-by: Deluan <deluan@navidrome.org> * review Signed-off-by: Deluan <deluan@navidrome.org> --------- Signed-off-by: Deluan <deluan@navidrome.org> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: deluan <deluan.quintao@mechanical-orchard.com>
This commit is contained in:
Deluan Quintão
@@ -43,5 +43,5 @@ type DataStore interface {
|
||||
|
||||
WithTx(block func(tx DataStore) error, scope ...string) error
|
||||
WithTxImmediate(block func(tx DataStore) error, scope ...string) error
|
||||
GC(ctx context.Context) error
|
||||
GC(ctx context.Context, libraryIDs ...int) error
|
||||
}
|
||||
|
||||
@@ -85,7 +85,7 @@ type FolderRepository interface {
|
||||
GetByPath(lib Library, path string) (*Folder, error)
|
||||
GetAll(...QueryOptions) ([]Folder, error)
|
||||
CountAll(...QueryOptions) (int64, error)
|
||||
GetLastUpdates(lib Library) (map[string]FolderUpdateInfo, error)
|
||||
GetFolderUpdateInfo(lib Library, targetPaths ...string) (map[string]FolderUpdateInfo, error)
|
||||
Put(*Folder) error
|
||||
MarkMissing(missing bool, ids ...string) error
|
||||
GetTouchedWithPlaylists() (FolderCursor, error)
|
||||
|
||||
81
model/scanner.go
Normal file
81
model/scanner.go
Normal file
@@ -0,0 +1,81 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ScanTarget represents a specific folder within a library to be scanned.
|
||||
// NOTE: This struct is used as a map key, so it should only contain comparable types.
|
||||
type ScanTarget struct {
|
||||
LibraryID int
|
||||
FolderPath string // Relative path within the library, or "" for entire library
|
||||
}
|
||||
|
||||
func (st ScanTarget) String() string {
|
||||
return fmt.Sprintf("%d:%s", st.LibraryID, st.FolderPath)
|
||||
}
|
||||
|
||||
// ScannerStatus holds information about the current scan status
|
||||
type ScannerStatus struct {
|
||||
Scanning bool
|
||||
LastScan time.Time
|
||||
Count uint32
|
||||
FolderCount uint32
|
||||
LastError string
|
||||
ScanType string
|
||||
ElapsedTime time.Duration
|
||||
}
|
||||
|
||||
type Scanner interface {
|
||||
// ScanAll starts a scan of all libraries. This is a blocking operation.
|
||||
ScanAll(ctx context.Context, fullScan bool) (warnings []string, err error)
|
||||
// ScanFolders scans specific library/folder pairs, recursing into subdirectories.
|
||||
// If targets is nil, it scans all libraries. This is a blocking operation.
|
||||
ScanFolders(ctx context.Context, fullScan bool, targets []ScanTarget) (warnings []string, err error)
|
||||
Status(context.Context) (*ScannerStatus, error)
|
||||
}
|
||||
|
||||
// ParseTargets parses scan targets strings into ScanTarget structs.
|
||||
// Example: []string{"1:Music/Rock", "2:Classical"}
|
||||
func ParseTargets(libFolders []string) ([]ScanTarget, error) {
|
||||
targets := make([]ScanTarget, 0, len(libFolders))
|
||||
|
||||
for _, part := range libFolders {
|
||||
part = strings.TrimSpace(part)
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// Split by the first colon
|
||||
colonIdx := strings.Index(part, ":")
|
||||
if colonIdx == -1 {
|
||||
return nil, fmt.Errorf("invalid target format: %q (expected libraryID:folderPath)", part)
|
||||
}
|
||||
|
||||
libIDStr := part[:colonIdx]
|
||||
folderPath := part[colonIdx+1:]
|
||||
|
||||
libID, err := strconv.Atoi(libIDStr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid library ID %q: %w", libIDStr, err)
|
||||
}
|
||||
if libID <= 0 {
|
||||
return nil, fmt.Errorf("invalid library ID %q", libIDStr)
|
||||
}
|
||||
|
||||
targets = append(targets, ScanTarget{
|
||||
LibraryID: libID,
|
||||
FolderPath: folderPath,
|
||||
})
|
||||
}
|
||||
|
||||
if len(targets) == 0 {
|
||||
return nil, fmt.Errorf("no valid targets found")
|
||||
}
|
||||
|
||||
return targets, nil
|
||||
}
|
||||
89
model/scanner_test.go
Normal file
89
model/scanner_test.go
Normal file
@@ -0,0 +1,89 @@
|
||||
package model_test
|
||||
|
||||
import (
|
||||
"github.com/navidrome/navidrome/model"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("ParseTargets", func() {
|
||||
It("parses multiple entries in slice", func() {
|
||||
targets, err := model.ParseTargets([]string{"1:Music/Rock", "1:Music/Jazz", "2:Classical"})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(targets).To(HaveLen(3))
|
||||
Expect(targets[0].LibraryID).To(Equal(1))
|
||||
Expect(targets[0].FolderPath).To(Equal("Music/Rock"))
|
||||
Expect(targets[1].LibraryID).To(Equal(1))
|
||||
Expect(targets[1].FolderPath).To(Equal("Music/Jazz"))
|
||||
Expect(targets[2].LibraryID).To(Equal(2))
|
||||
Expect(targets[2].FolderPath).To(Equal("Classical"))
|
||||
})
|
||||
|
||||
It("handles empty folder paths", func() {
|
||||
targets, err := model.ParseTargets([]string{"1:", "2:"})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(targets).To(HaveLen(2))
|
||||
Expect(targets[0].FolderPath).To(Equal(""))
|
||||
Expect(targets[1].FolderPath).To(Equal(""))
|
||||
})
|
||||
|
||||
It("trims whitespace from entries", func() {
|
||||
targets, err := model.ParseTargets([]string{" 1:Music/Rock", " 2:Classical "})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(targets).To(HaveLen(2))
|
||||
Expect(targets[0].LibraryID).To(Equal(1))
|
||||
Expect(targets[0].FolderPath).To(Equal("Music/Rock"))
|
||||
Expect(targets[1].LibraryID).To(Equal(2))
|
||||
Expect(targets[1].FolderPath).To(Equal("Classical"))
|
||||
})
|
||||
|
||||
It("skips empty strings", func() {
|
||||
targets, err := model.ParseTargets([]string{"1:Music/Rock", "", "2:Classical"})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(targets).To(HaveLen(2))
|
||||
})
|
||||
|
||||
It("handles paths with colons", func() {
|
||||
targets, err := model.ParseTargets([]string{"1:C:/Music/Rock", "2:/path:with:colons"})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(targets).To(HaveLen(2))
|
||||
Expect(targets[0].FolderPath).To(Equal("C:/Music/Rock"))
|
||||
Expect(targets[1].FolderPath).To(Equal("/path:with:colons"))
|
||||
})
|
||||
|
||||
It("returns error for invalid format without colon", func() {
|
||||
_, err := model.ParseTargets([]string{"1Music/Rock"})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("invalid target format"))
|
||||
})
|
||||
|
||||
It("returns error for non-numeric library ID", func() {
|
||||
_, err := model.ParseTargets([]string{"abc:Music/Rock"})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("invalid library ID"))
|
||||
})
|
||||
|
||||
It("returns error for negative library ID", func() {
|
||||
_, err := model.ParseTargets([]string{"-1:Music/Rock"})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("invalid library ID"))
|
||||
})
|
||||
|
||||
It("returns error for zero library ID", func() {
|
||||
_, err := model.ParseTargets([]string{"0:Music/Rock"})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("invalid library ID"))
|
||||
})
|
||||
|
||||
It("returns error for empty input", func() {
|
||||
_, err := model.ParseTargets([]string{})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("no valid targets found"))
|
||||
})
|
||||
|
||||
It("returns error for all empty strings", func() {
|
||||
_, err := model.ParseTargets([]string{"", " ", ""})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("no valid targets found"))
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user