Streaming music from cloud storage

[GrumpyPatzer]

Thanks for the quick and comprehensive reply.

(25-11-2023 18:44)simoncn Wrote:  I will be interested to hear whether you are able to find a workable cloud-based solution. Ideally this would look like a network drive mounted via an SMB connection.

(25-11-2023 17:47)GrumpyPatzer Wrote:  1. When MinimSever scans a music library, is it able to access an individual file’s tag information without loading the whole file?

MinimServer reads only as much of the file as is needed to read the audio information and tag information. For most file formats, this information is at the start of the file. In a few cases, tag information can be at the end of the file. If so, MinimServer seeks to this point in the file rather than reading all the intervening data. How well this seek performs depends on the file system.

Quite a lot of work has gone into fine-tuning this.

Quote:2. Can MinimServer persist the information it obtains from scanning the music library across reboots?

When the first scan has completed, all the information that has been read is stored in a local cache file on disk. A subsequent rescan reads this cache file in tandem with scanning the library. For each file in the library, if the file size and last modified date match those in the cache entry for the file, the cached information in loaded and no data is read from the file. Any new library files not found in the cache are read (as described above) and their file data is added to the cache. Any files in the cache but no longer in the library have their cache data deleted. Rescanning in this way is typically about 10x faster than the initial scan. Again, quite a lot of work has gone into fine-tuning this.

There is no automatic rescanning in MinimServer. After any change to the library, the user must rescan manually. This is different from Twonky, which spends a lot of CPU resource watching for file changes and rescanning when it detects a change.

Rather neat (on both counts!). As it turns out, these optimisations may prove to be very relevant to streaming from cloud based storage. I could not resist having a shot at getting this working (having already uploaded about 25% of my music collection, so roughly 250GB, to MS OneDrive). In no way could I claim to have thought carefully about how best to do this; all I have done so far is to google ways to effect a local mount of OneDrive storage and to spend a few minutes experimenting with different options. There is undoubtedly scope to tune the performance (it is most unlikely that the configuration I am using is optimal). And I strongly suspect that there will be other cloud storage services that will be better suited to streaming media.

That said, I have got it working, to the point where I have (a) scanned just over 8,000 FLAC files; and (b) been able to play several pieces, including a 192/24 recording, with no drop-outs or undue delays. This is running MinimServer v2.2 on a MacBook Pro with a 100Mbps FTTP broadband connection, using an old Linn DS player with a wired LAN connection. The two packages I am using to make the OneDrive storage appear as a locally mounted part of the filesystem are called rclone and FUSE-T.

Some observations:

1. As I anticipated, the initial scan is slow. Having spent only a few short time tuning, the scanning rate I achieved was c.3 files per second (although I now realise that that was with the laptop connected using wifi, not an ethernet cable, which may have introduced some further latency).

2. When I first viewed the library on my iPad, loading the artwork for 274 albums was not particularly snappy (it took a second or two each time I scrolled down), but that was far from unusable. And I am wondering if there is some caching of artwork that will then come into play.

3. I have only played about a quarter of an hours’ worth of music, but it worked flawlessly. No drop-outs or problems of any kind. It is possible that, after playing music for longer, OneDrive might start throttling (if it thinks that too many API requests are being made), but that remains to be seen.

4. For this to be a good alternative to a NAS, I would want it to be easier to use than it currently is. With my NAS, I simply turn it on and off. But it may be possible to achieve a similar level of user friendliness with a Raspberry Pi (particularly if one is prepared to leave it turned on all the time; I certainly would not wish to do that with a laptop).

Now, it may be that there is scope to improve scanning performance somewhat. The first thing to look at, I think, is the set of rclone parameters that control things like buffer sizes and caching. Rather strangely, the best scan rate I have achieved so far has been with a FUSE-T rwsize of 2MB and rclone ifs-read-chunk-size and buffer-size set to 512kB.

In isolation, those parameters don't mean much (I certainly would not claim to know precisely how rclone and FUSE-T work). And I’m not expecting advice about what might work better. But, in order to have some chance of tuning it, I think I need to better understand how MinimServer’s scanning process works. In the MinimServer log, I have observed trace records like

FLAC block type-0 length=34
FLAC block type=4 length=967
Found Vorbis comment block
FLAC block type=3 length-216
FLAC block type=-127 length=7265

The length of the type 0 blocks always seem to be 34, the others vary a bit.

So, if you’ll forgive a couple more questions (sorry to be a nuisance …):

1. With FLAC files like this, how much data is MinimServer having to read, in order to obtain the tag information?

2. Is this reading of tag information being done using several (four?) distinct read operations?

(Depending on how exactly rclone and FUSE-T work, it may be that each distinct read operation has some associated latency, even if the data has been buffered.)

And one further question (which may seem unrelated but is relevant):

3. Having used it on one system, is there a way to transfer that instance of the MinimServer license to another system?

(The reason that is relevant is that I would like to try this cloud storage-based operation on a Raspberry Pi. I have ordered a new one, but might try it on a much older model if it’s possible to then switch the license to the new Pi when it arrives. But it’s absolutely no problem at all if the license is fixed to the system on which it is first used; if that’s the case, I will simply wait until the new Pi turns up.)


Thanks ….
Andrew