Best Way to Back up Music Library

[DavidL]

Best of luck with your backup!

For any general readers of this thread it might be of interest to describe an alternative approach to library update / backup that I've been using successfully for a year.

I also store my music library on a Synology NAS on which MinimServer and MinimStreamer run to play the music via an ethernet network.
I make and update 2 backups. For the prime, local, backup I use the Synology backup application to copy the library to a separate hard disk connected by an eSATA port. For a secondary backup I use another hard disk connected to a computer running ChronoSync software to periodically backup across the ethernet network. I store the latter disk remotely to avoid any common-mode failure effects (lightning strike, flood etc etc).
To update the library I use iTunes running on a computer on the network which accesses all the iTunes files (library management files and media files) held on the Synology NAS. I do not use iTunes for music playback.
This relatively simple system was easy to set up and has worked without any problems.

[gnomus]

(03-06-2015 16:09)DavidHB Wrote:  ...
Good luck. As you are starting from scratch, in the worst case you can always delete and start again. Don't hesitate to ask about any issues that arise.

David

I will update for those who may be thinking of using SyncToy and who may, like me, not be all that savvy with these things.

It was quite interesting running the program - I found that there were many more files that needed "overwriting" than I expected (around a 1,000). At first, I thought these must be corrupted in some way, until I worked out why they were being updated...... Up until now, when I have discovered minor tagging errors, I have corrected the errors separately in each of the 4 locations that I have saved the music library. So if I had to change something in the "Life on Mars.flac" file on the Synology, I would also have to change the same thing on the hard disks where my library is copied. Of course, this mean that, even though the "music" in the file is identical, the files themselves are not the same since the time they were last altered is different (by a minute or so). As far as I could tell, all of the files that needed to be overwritten were ones I had had to make tagging corrections to.

I like the fact that you can use SyncToy to "Preview" changes that would be made if you were to run the utility. I did this to compare the libraries on my Synology and my QNAP. Here, I found a lot of strange things. I found a lot of odd directories on the QNAP, some of which contained files. Virtually all of my music folders contain an @ea folder. Like this:

   

Inside this @ea folder I found:

   

As you can see, there is a folder for each of the tracks on the Magical Mystery Tour album. Within those folders there were jpg files. Here, for example, are the contents of the Strawberry Fields folder:

   

When I opened these, oddly named, jpg files, I found that they were simply the Magical Mystery Tour album art.

I don't know how to account for these - and I have many of them (thousands). I do recall that at one stage, before someone here (Simon) showed me the light, I was ripping CDs directly to the Synology, and then copying from there to my back up locations (such as the QNAP). I don't know if that might be the cause. I also don't know if I should work my way through my QNAP library deleting all of these @ea directories and their contents. Some of them are several megabytes in size

Does anyone have any thoughts?

[DavidHB]

(04-06-2015 06:55)gnomus Wrote:  I will update for those who may be thinking of using SyncToy and who may, like me, not be all that savvy with these things.

Thank you for posting your results. I'm glad that SyncToy seems to be working for you.

SyncToy is not that technical a program to use; certainly it should not be much of a challenge for anyone who can work MinimServer . The one decision which can be difficult for a new user is which 'action' to use - synchronise, echo, or contribute. For the reasons you have now demonstrated rather well, I think that the easiest thing for a new user is to have only one master/source folder, make sure the contents of that folder are what is required, and then echo (= mirror) that folder to the backup or slave locations.

(04-06-2015 06:55)gnomus Wrote:  It was quite interesting running the program ... As far as I could tell, all of the files that needed to be overwritten were ones I had had to make tagging corrections to.

It sounds as though things have worked exactly as they should. Your account took me back. I got into digital photography well over a decade before I became interested in streaming music, so I originally used SyncToy for my photo collection. My experience then was similar to yours; folders I had thought were well organised had all kinds of files I did not know were there. I decided to back up the whole of my messy library as it was, and cleaned it up to become the master.

The cleaning process was effective; in the event, I never needed the backup, and I am still using the same master library, though (thanks to SyncToy) it has been moved from computer to computer several times during the intervening period. When I decided to build a digital music library, I used the same system, and it works just as well with the music as it does with the images.

(04-06-2015 06:55)gnomus Wrote:  I like the fact that you can use SyncToy to "Preview" changes that would be made if you were to run the utility. I did this to compare the libraries on my Synology and my QNAP. Here, I found a lot of strange things ... I don't know how to account for these - and I have many of them (thousands). ... I also don't know if I should work my way through my QNAP library deleting all of these @ea directories and their contents. Some of them are several megabytes in size. Does anyone have any thoughts?

I don't know where these files came from (often, you never do know); because generating thousands of files without the user's knowledge requires a lot of background access, my guess would be is that the culprit is whatever media server comes pre-installed on the QNAP. If you don't use this server for music, the files are presumably surplus to requirements. If you happen to have the space (say an odd hard drive you are not using), you could do what I did with your photos, and create an offline backup of the QNAP music folder, just in case there is an issue. Any files on the QNAP that you want to keep should be copied to the master library (on the 213J, I think) if they are not already there. Then you could use SyncToy to echo the master library to the music folder on the QNAP. That process will remove all the spurious files as well as any other inconsistencies, and the backup is there if they turn out not to be spurious after all. If you can't stop the QNAP application, it would presumably go on generating files, but at least these would be deleted every time you used SyncToy.

Thank you also for mentioning the SyncToy preview, which I should have referred to as it is a very good safety feature.

Going forward, the name of the game is to maintain consistency between all the copies of your library, which is what having a single master folder (where you do all your ripping and metadata changes) and the echo action of SyncToy are all about. As a mentor said to me many years ago, "If you cannot be consistently right, you should at least be consistently wrong".

David

[gnomus]

Thanks again for all your help David and for pointing me towards SyncToy. I will start using it for my photographs too.

I have deleted a few of the @ea folders from the QNAP, and it seems to make no difference to the music on the QNAP library. I am thinking of doing a search for all @ea folders and simply deleting them. I don't know if I quite have the courage, though. In the meantime I am 'echo-ing' all of the other back-up libraries. That will keep me busy for a day or two.

[Edit: By the way, I agree that SyncToy is very straightforward to use. I really know very little about these things, but, once I saw how it worked, I got quite a bit of confidence with it. I agree that 'echo' is the method to use.]

[simoncn]

(04-06-2015 06:55)gnomus Wrote:  I don't know how to account for these - and I have many of them (thousands). I do recall that at one stage, before someone here (Simon) showed me the light, I was ripping CDs directly to the Synology, and then copying from there to my back up locations (such as the QNAP). I don't know if that might be the cause. I also don't know if I should work my way through my QNAP library deleting all of these @ea directories and their contents. Some of them are several megabytes in size

Does anyone have any thoughts?

I believe these are thumbnail images that are created automatically by something in Synology DSM. You don't see these folders on the Synology because Synology DSM treats them as hidden folders. MinimServer doesn't use these images and there is no problem with deleting them.

[gnomus]

Fascinating. The preview I ran had the QNAP as the 'left' folder. If these files originated on the Synology (and I'm sure Simon is correct), then it is interesting that SyncToy clearly couldn't see them on the Synology, since it wanted to copy them back from the QNAP to the Synology.

If I were to 'echo' the Synology to the QNAP, would these files be deleted, or would I only end up with a new load of @ea directories, I wonder?

In terms of getting rid of them on the QNAP, is the best thing just to do a search for them using Windows Explorer, do a 'Select All' and then a 'Delete'?

[Edit: By the way there were over 14,000 of these @ea directories.]

[DavidHB]

I've been doing some web searching, and I think that Simon is correct; the folders originated on the Synology. The "ea" bit apparently stands for "extended attributes".

I can easily see how folders which are hidden on the Synology could become visible on the QNAP, especially if their hidden status is in any way dependent in the extended attributes they contain. For its part, SyncToy, in the settings you are using is copying files with all attributes, and the Synology has probably hidden its own copies of the "@ea" folders from SMB/Windows and therefore from SyncToy.

In the circumstances, I would delete these folders from the QNAP, where they apparently do not belong, before synchronising with the Synology. For future runs of any SyncToy echo task with the Synology as left folder, I would set an exclude mask in the task properties, so that these files are not copied, and I would also exclude hidden files from the task. Both these things are easy to set in the options dialog.

Unfortunately, problems of this kind will be an issue with any file synchronisation program. But at least you can see that the caution with which you approached the task was entirely justified.

David

[gnomus]

Thanks again David. I'll have a look at those options.

I think I may be getting to the bottom of why the 213J failed in the first place. I have 2 x 3TB drives in my 213J. It failed - by which I mean it was nowhere to be seen on my network and it kept making a beeping noise and showing an orange status light). Someone from Synology suggested that I reset the NAS and then plug it directly into an ethernet port on my computer. This seemed to allow me to see the NAS again. However, it reported that there was a problem with "Volume 1" (which I took to be Hard Disk 1). It suggested that it could repair this volume and I asked it to do that for me. Some 12 hours later it seemed to have worked and the volumes/disks were reported as being "Good". Today I am getting an amber light on Disk 1 again and, when trying to run SyncToy the Synology is reporting "I/O error occurred to hard disk 1 on DiskStation". It is looking as if that drive is kaput therefore. I have ordered a new HD (Western Red). Do I just remove the defective disk and insert the new one, or is it a little more complicated than that? The 213J was set up so that the data was mirrored in some way (I don't really understand this RAID business). Should the original data re-write to the new disk when it is inserted?

Thanks again - jings I'm having a lot of issues at the moment..

[DavidHB]

@gnomus

My NAS is not a Synology, and I've never had a RAID array drive fail on me, so I cannot claim this as an area of expertise. Simon does have at least one Synology NAS, so he may be a better source of advice.

Obviously, you will need to remove and replace the hard drive. Unfortunately, I don't know how automatic the recovery procedure is when you restart the NAS; I'd guess that it is a lengthy process (some hours).

RAID comes in different varieties, and most manufacturers have variants of their own, so you will need documentation specific to your NAS. If you don't still have the paper manual (or there never was one), you can get a PDF copy from the Synology website. I've had a quick look, and the manual seems to be suggesting that the procedure for recovering a degraded RAID array (their term) is to be found in DSM Help.

It certainly does not surprise me that a hard drive on its last legs might tend to fail under the stress of a heavy duty copying operation. While I see RAID as a useful extra layer of protection, I don't think that it is a sensible alternative to an effective backup routine.

David

[gnomus]

I speak from a position of ignorance, of course, but, some time back, I did see a post (I think from Simon) that pointed out that the second drive in did not provide a 'back up' as such - if a file became corrupted it simply meant that the corrupted file was duplicated.

It is a little expensive buying all these second drives, and I am wondering if I might be better off just having singles in my Synology - or setting it up as one large 6TB storage device. I'd be interested in what folks thought of this idea.