Rescan Speed

[lyapounov]

(27-04-2021 15:35)simoncn Wrote:  If it takes 3 minutes to do the rescan, this is about the same time as it takes to play a song. You can start a song and then start the rescan. The song will continue to play while the rescan is happening. By the time the song has finished, your new playlist should be available.

Maybe one day my music library will be so big that I will have to play the entire Ring before the scan is finished

[usery]

(28-04-2021 18:57)simoncn Wrote:  Thanks for your understanding. I am hoping to improve the speed of the use case you have described but this will require major redesign of some of the Minimserver internals with careful thought to ensure it doesn't affect browsing speed or memory usage.

Any progress improving rescan & metadata rebuild speed on incremental library additions?

On minimserver v2.1 update 216 with startupscan = false, adding just one or two new directories (< 500MB of .flac files) and then 'rescan' will on occasion cause a rescan of my entire library of 2.7TB (> 178,000 files in ˜10,800 directories), lasting up to 30 minutes. Is that expected for a library of this size?

[mike963]

If you have a lot of playlists - I found out that my playlists (I had a lot and they contained a lot of tracks) made the performance of any sort of scan really slow so I ended up moving my playlists from server to client

[simoncn]

Rescanning about 180,000 files in about 11,000 directories should take about 3 minutes plus the time taken to read any new files (usually a few seconds). If this takes 30 minutes, it sounds like MinimServer is rebuilding its cache file. Does a rescan always take 30 minutes or does it sometimes take 3 minutes?

[usery]

(27-06-2022 09:41)simoncn Wrote:  Rescanning about 180,000 files in about 11,000 directories should take about 3 minutes plus the time taken to read any new files (usually a few seconds). If this takes 30 minutes, it sounds like MinimServer is rebuilding its cache file. Does a rescan always take 30 minutes or does it sometimes take 3 minutes?

Most of the time rescan is fast (< 1min) after adding a small number (≤ 10) of new files, but occasionally and maddeningly it's up to 30min. No discernible pattern for when it 'decides' to rebuild cache / metadata.

fwiw I have no playlists.

[simoncn]

Could the longer rescans perhaps be after you have made some change to the MinimServer configuration such as adding new entries to indexTags? This could require a full rescan.

Another possibility is that the MinimServer cache file might somehow have been deleted. What device are you using to run MinimServer?

[usery]

(27-06-2022 17:15)simoncn Wrote:  Could the longer rescans perhaps be after you have made some change to the MinimServer configuration such as adding new entries to indexTags? This could require a full rescan.

Another possibility is that the MinimServer cache file might somehow have been deleted. What device are you using to run MinimServer?

No changes to MinimServer configuration, ever. It's seemingly stable so I don't mess with it.

MinimServer is running on Win10 64 Pro - standard config, no weirdo services or background processes afaict. Library is on an internal volume.

The 30min rescans happen every 1-2 out of 10 rescans.

[simoncn]

There is a small chance that for some reason MinimServer has encountered an error reading the cache file. Next time this happens, please check the MinimServer log to see if there are any error messages that might indicate this.

[Chopzee]

Hi,
I used to have a slow rescan on my DS218+ until I increased the memory on the NAS. I have 280,000 songs a mixture of MP3 and Flac. Unless I add a large number of files at once (500 or so) a rescan takes no time at all. Is this because the cache on the NAS is holding the data from the scan? I have noticed if I haven't rescaned for a few days then the scan does then take a lot longer, but not as long as before I added the extra memory.
I run the scan remotely from my PC using minimwatch.
Regards

[simoncn]

Rescanning a library of this size needs a significant amount of memory. If you don't have enough RAM for this, the operating system will need to do swapping to/from disk which will make the rescan slower.

A rescan checks all files to see if they have changed and reads any files that have been added or changed. The time taken for checking should be the same for every rescan (but see below). The time taken for reading will be greater if there are more added or changed files to read.

If you haven't done a rescan for a few days, relevant data in the operating system cache and/or disk cache will have been overwritten and will need to be recreated. This is why the rescan takes a bit longer in this case.