Rescanning Estimate

[paultaylor]

I am in process of adding an option to SongKong to allow the user to rescan MinimServer after FixSongs task has completed so its picks up modified metadata. I have it working using the webapi HttpPost method.

I make it optional tickbox because may not want to do it every time time due to the time taken to rescan. But I notice when rescan it is actually very quick even on large libraries if only a few files were modified, so am I correct in thinking MinimServer only rescans those song files that have been modified (via checking the file modifiedDate), and this reading of file contents takes the majority of the time, not the browsing of folder structure to identify music files ?

Then it would be nice if I could inform user of progress and completion of rescan. I can use the about command to identify when rescan has completed. However I can only poll this to find out when completed, and I have no idea how long the rescan would take. Would be very useful if MinimServer could estimate how long task will take, and allow application to query minimserver to see progress (perhaps as percentage).

[simoncn]

The time taken to rescan has three components:

1) Reading the cache file
2) Checking the attributes of each previously scanned file in the library to find out if it has been modified and needs to be read again
3) Reading all modified files (identified in step 2) and all files added since the last rescan to obtain their audio data and tag information

In theory, MinimServer should be able to estimate 1 and 2. It cannot estimate 3 because it doesn't know how many existing files have been modified and how many new files have been added since the last scan. As this is the main component of rescan time, any estimate by MinimServer is likely to be very inaccurate.

However, SongKong presumably does have this information and should therefore be well placed to estimate the rescan time. This could be based on how long the last rescan took with a suitable adjustment for the number of modified and added files.

[paultaylor]

Going back a step, in the past there was issue that MinimServer had to rebuild complete cache, is that still the case or not ?

It sounds like you only have to reread modified files not all files, therefore if reading the files is the the slow bit rebuilding the complete cache is not a big deal ?

What I am getting at is rescan quicker than it used to be or not?
If only change a few files (and no new files to add) does that mean rescan will be quick and therefore no problem running rescan each time?

My understanding was if doing some modification on individual albums one a time, then best to do one rescan at the end, but if the rescan would be quick then seems no real problem running it each time. If modified many files then rescan will take longer but really needs to be done for control point, so again makes sense to run rescan ?

[simoncn]

There have been no changes to how rescan works since we last discussed this. It still rebuilds the cache and it still makes MinimServer unavailable in control points while the rescan is happening (perhaps 2 to 3 minutes for a moderately large library with a small number of changed files, or longer if running MinimServer on a slow device). It is also possible that the control point could lose its current position in the MinimServer browse tree when it rediscovers MinimServer after the rescan.

For all these reasons, it would not be a good idea for SongKong to automatically initiate a rescan after a single change. It is better for the user to complete a set of changes and manually initiate a rescan when the user has completed their changes and wants to see them in MinimServer.

[paultaylor]

(07-12-2023 11:06)simoncn Wrote:  It is better for the user to complete a set of changes and manually initiate a rescan when the user has completed their changes and wants to see them in MinimServer.

Thanks for the summary, it will not be automatic, but important to make easily available from SongKong because it is still a common problem that after using SongKong the changes do not show in customers control point and they dont understand why.

Back to rescanning estimate, it is true that SongKong knows how many files have been modifed by SongKong, and also how many files have been renamed (I dont know if this is a new file or not by MinimServer but I think it maybe treated differently if running on unix system to windows system). It wouldn't know about any changes made by any other application but Ill ignore that complication for now.

However, even if we estimate time taken to rescan there is no way to check progress, all we can do is keep polling MinimSserver to see if it has finished. Users need to see a progress bar so they have confidence it is progressing, I could create a progress bar and increment based on my estimate but unless I get it just right the progress bar would reach 100% too late or too early.

Any estimate that can be done by SongKong could be better done by MinimServer, even if estimate progress time cannot be accurately calculated immediately it would be possible to make an estimate and revise that estimate as you go along. i.e once read cache know how may songs have been modified.

This would be a improvement to MinimServer, currently when using the Web UI you can select the Rescan button and status changes to Restarting. But it doesn't give any indication how long it would take and you don't know when it has completed without periodically selecting Refresh to check

[simoncn]

(07-12-2023 13:31)paultaylor Wrote:  Back to rescanning estimate, it is true that SongKong knows how many files have been modifed by SongKong, and also how many files have been renamed (I dont know if this is a new file or not by MinimServer

A renamed file is treated as a deletion and addition on all platforms.

Quote:I could create a progress bar and increment based on my estimate but unless I get it just right the progress bar would reach 100% too late or too early.

Exactly the same applies to any progress bar that MinimServer could create (see below).

Quote:Any estimate that can be done by SongKong could be better done by MinimServer, even if estimate progress time cannot be accurately calculated immediately it would be possible to make an estimate and revise that estimate as you go along. i.e once read cache know how may songs have been modified.

The cache is read in tandem with scanning the library, so MinimServer does not know how many files have been modified, deleted and added until all files in the library have been scanned. SongKong knows this before the scan starts so it is better able to make this estimate.

Quote:This would be a improvement to MinimServer, currently when using the Web UI you can select the Rescan button and status changes to Restarting. But it doesn't give any indication how long it would take and you don't know when it has completed without periodically selecting Refresh to check

If MinimWatch or the MinimServer native GUI is being used, the minim icon changes colour to indicate completion. I am looking into enabling the MinimServer configuration web page to update asynchonously to provide this notification. At present I don't see a simple way to provide this notification via the command scripting interface.

[paultaylor]

(07-12-2023 14:05)simoncn Wrote:  Exactly the same applies to any progress bar that MinimServer could create (see below).

No thats incorrect because Minmserver internally knows how it is progressing with the rescan so it can adjust its estimate but SongKong does not and cannot. You could modify rescan to take an argument which is is how many files the calling program expects to be modified but it would only be an estimate and I dont think it should be necessary.

Quote:The cache is read in tandem with scanning the library, so MinimServer does not know how many files have been modified, deleted and added until all files in the library have been scanned. SongKong knows this before the scan starts so it is better able to make this estimate.

Previously it sounded like you used reaching of the cache file to decide what songs to update so I'm not clear why you are reading the cache file if it isn't doing that. If it is doing that I expect the read of the cache occurs very quickly. It should be possible to structure MinimServer so it can do this even if cannot do it currently.

But my general point is that it always make sense for any application itself to estimate task duration,and provide progress update on task. It does not make sense to expect a calling application to make estimate, and it is impossible for calling application to provide actual progress update.

Quote:I am looking into enabling the MinimServer configuration web page to update asynchonously to provide this notification

That is an improvement but this still would not give the user any indication of when it is expected to finish. They would still have to periodically look at the webpage to check if finished without any idea if nearly finished or not, they just wouldn't have to click on the refresh button.

[simoncn]

The cache file is read in tandem with the library scan, so the cache is read for one folder, then the folder is processed, then the cache is read for the next folder, etc. Early folders might be processed very quickly with few or no changes and later folders might be processed very slowly with many changes or new files, or vice versa. MinimServer does not read ahead and cannot predict what will happen later in the scan.

We have all seen progress bars that move up to 99% and take forever to get to 100% or that stick at 20% for a long time, then race ahead to 70%, then stall again, etc. Adding this type of progress bar to MinimServer does not seem very useful.

I think we will need to leave this unresolved for now and consider it again when I am able to work on the major changes for improving rescanning speed that I would like to implement.

[paultaylor]

(07-12-2023 17:54)simoncn Wrote:  I think we will need to leave this unresolved for now and consider it again when I am able to work on the major changes for improving rescanning speed that I would like to implement.

Okay, thanks I think it will be useful when can be accurately implemented.

[MiniMe]

Why not implement minim server rescan option with a hourglass or spinning wheel icon to show that it is rescanning. It will be back before you know and once used to the time it takes it's not needed to show a progress bar, at least not for me