lib/model: Don't pull if ignores failed to load and cleanup (#5418)

This commit is contained in:
Internal Server Error - Gitea: Git with a cup of tea
500 Internal Server Error

Gitea Version: 1.28.0+dev-477-g8b6ad49a5f