Recently added files

[paulM]

Simon

Yesterday I did a rescan after adding 3 new albums but the recently added files showing in my control point have inexplicably become jumbled up. Does Minimserver do backups of the recently added files because I would like to go back to the output of Friday's rescan to get the recently added files back into the correct order?

[simoncn]

MinimServer doesn't do automatic backups of the recently added information files.

The first step is to find out what caused the problem. It might be possible to edit the recently added information files to correct it. Before doing any more MinimServer rescans or restarts, copy the added-dates.txt and added-files.txt files to a safe location and attach these files to a post here (don't copy the contents inline).

If you aren't sure where these files are located, please post details of which platform (QNAP, Synology, Windows, etc.) you are using to run MinimServer.

Please also post details of which version of MinimServer you are running, including the update level.

[paulM]

I am running Minimserver 2.2, update 252, on a QNAP HS251. I attach a copy of added-dates.txt for your information.

Lines 1 to 66 appear to be the only part of the original sequence to remain. The last successful rescan occurred on 2024-06-14 at 17:26 and this rescan ends at line 66.

The problematic rescan occurred on 2024-06-15 at 17:05. It starts off fine and the first 2 of 3 albums were added between lines 67 and line 99. Everything from that point is out of sequence. You will note that the last album in correct sequence (Richard Thompson - Electric) is then followed by a dozen other Richard Thompson albums, all of which have been on the server for years. There then follows a number of other albums that have been on a long time until you get to line 840, when the third of the three "new" albums is added. You will note that the third recently added album (U Roy - Pray fi di People on lines 840 to 852) is preceded by another U Roy album that has been on for a significant length of time. I don't know if this is significant, but it is an observation. The third of the three new albums ends at line 852.

Line 853 to line 999 are all recently added albums, but due to what happened above, they are significantly out of sequence.

Line 1000 makes me curious because it is an old .mp4 that has been on the server for over a decade. I'd forgotten all about it because it was an experiment that I did not follow up on. It is on the same drive as the music, but is not in the music directory and I have never noticed Minimserver identify this file before. It is certainly not recent though!

Given that everything was ok up to and including the album "Electric" and goes very wrong immediately after this, I am guessing that this album is the cause of my woes. Just in case it is, I need to observe that this album has existed on my server for quite a few year as a single disk .mp3 version. On Saturday, I removed the .mp3 version and replaced it with the deluxe version in .flac format. So it changed format and an additional bonus disk was added. I don't think that this is the issue as I have upgraded other albums in the past without problem when I have sourced better versions - but I mention it for completeness.

I think that this about covers it.

[simoncn]

Thanks for posting this file along with your very helpful detailed description of exactly what is wrong.

I am aware of a scenario in which MinimServer can incorrectly detect files as having been added since the last scan. This scenario is as follows:

1) A scan is performed which ends early (the whole library is not scanned). No error indication is received by MinimServer that this has happened. This appears to be an issue with either the NAS firmware or the Java runtime.

2) Because the scan ends early, the MinimServer cache file is incomplete. Also, the library contents are incomplete. This is a clue that all is not well but it could easily be missed.

3) The user does another scan. All library files scanned in this scan that were not scanned in the previous scan are incorrectly detected as newly added because they aren't found in the cache file written by the previous scan.

4) MinimServer adds these incorrectly detected files to the added-dates.txt file and deletes older files to stay within the maximum files limit (1000 in your case).

This issue has been reported by one user and I had hoped it was a one-off occurrence. I am tempted to put your problem down to this issue but there is a factor that makes me think this is not the same problem. This factor is the other Richard Thompson albums that were detected as newly added immediately following the genuinely added album. It seems unlikely that this is a coincidence.

I will look carefully at the scanning code to see if I can see any way this could have happened. Please confirm whether you had already installed update 252 at the time of the problematical scan (2024-06-15 at 17:05) as this update has a small change to the scanning logic intended to work around some effects of the scenario I have described.

If you don't have any backups of the added-dates.txt file, the best you can do to recover from this problem is to delete lines 100 to 839 and 853 to 1000 of the added-dates.txt file (i.e., all incorrect lines) and then restart MinimServer. After the restart, you will see some older files in the Recently Added list and the added-files.txt file. These older files are selected based on "last modified" timestamps which should be fairly close to the dates they were added by a MiniMServer rescan.

[paulM]

I can confirm that update 252 was installed at the time of the problematic rescan on 15th June, as it was on the last successful rescan the previous day.

[simoncn]

Thanks for confirming this.

[paulM]

I have a bit of an update on this one. Thanks for the suggestion. I tried it but it didn’t work. By the look of the output, it looks like it would have worked but for one thing. I’m a bit of a fiddler when it comes to tags and have been correcting various spelling mistakes whenever I spot them but more significantly have been editing date fields where needed from four digit YYYY format to eight digit YYYY-MM-DD format, because when I browse album artists, I like to view them in chronological order rather than in alphabetical order. Using the four digit format, albums appear in alphabetical order when there are more than one album per year and this is the wrong order in a lot of cases. Unfortunately, I edited somewhere between 40 – 50 albums in mid-May from four digit to eight digit (they have probably been in the collection for a decade or so) – and these albums appeared in the middle of the recently added albums.

The sequence looks good both before and after these albums, but if the recently modified field is the one that Minimserver uses to recreate the recently added albums list, then it has done the right thing, but sadly, that is not what I want.

Consequently, I have another question. My music server backs up my collection every night onto a separate data server. It’s only there for backup purposes but a while back I installed my second licenced copy of Minimserver as a “just in case” implementation. I haven’t used it much but do a bit of experimating with it from time to time. The recently added files sequence on this implementation is imperfect because unlike my usual setup, I do not rescan with each new album. It can be weeks or months between rescans. But all of the last 1000 items in the list are the correct last 1000, even though the sequence is not quite perfect.

Will the files from the data server implementation work on the music server or will I do more harm by trying (I don’t like experimenting on my primary set up because I’ve screwed up in the past!)?

One final thing is that I have been experimenting with the data server implementation. I tried the same update with the same 3 albums and the data server recently added files screwed up as well. Not a biggie this time because I had backups to use to revert to the pre-rescan position. This got me thinking that maybe there was something wrong with one or other of the albums. So I tried to isolate which album it could be by rescanning one at a time. But by doing it this way, all 3 were added one by one without problem. So I continue to be baffled about what is happening. If they are of interest, I have the recently added files from before and after the failure on the dataserver.

But my main question is – will the recently added files from one implementation work on another?

[simoncn]

You can move the added-dates.txt file from one system to another. You might need to edit the file paths if they are not the same on the two systems. It should be possible to do this with a global find and replace operation in a text editor.

When MinimServer creates its recently added list, it starts with the contents of the added-dates file (using the dates in this file) and fills any unoccupied slots with the files that have the most recent "last modified" timestamps. This is why you are not seeing the correct order.

I would very much like to understand what has caused the problem with the 3 added files so that I can fix this problem. It is clearly not a random glitch. Would you be willing to set up a test to reproduce this error scenario and send me a log file? The log file would need to have debug logging enabled. If you are willing to do this, I will post detailed instructions.

[paulM]

I'm not clever enough to set up different file paths. The backup file stucture is identical to the master and minimserver has been set up on both NAS's using the defaults. The only difference is the name of the source (Dataserver as opposed to Mediaserver). So I'll give that a go.

With respect to testing - I'm more than happy to. I probably won't understand the answer, but this doesn't stop me from being curious!

By the way, I've done hundreds of rescans over the past couple of years without the slightest problem. I have noted that some contributors to this group have had speed issues, but Minimserver has been like the proverbial grit off a stick for me. The one and only issue that I have had with Minimserver was self inflicted (and that was being silly with the recently added files as well!).

Thanks for your help.

[paulM]

Just a quick note to confirm that the Dataserver recently added files worked fine on the Mediaserver. It's not 100% perfect, but it's pretty close. Thanks for that.