Streaming music from cloud storage

[GrumpyPatzer]

With apologies for having taken so long to reply ....

I finally found time to install rclone and MinimServer on my new Pi, and have tried playing some music using the copy of my collection that I have uploaded to MS OneDrive.

In summary: so far, it seems to work fine! See further comments below.

(30-11-2023 11:07)simoncn Wrote:  

(29-11-2023 21:19)GrumpyPatzer Wrote:  1. In both cases (command line test and MinimServer scan), it is typically taking between 200 and 350ms for each file operation, usually over 250ms but occasionally spiking to over 500ms. Unsurprisingly, the bulk of that time is that wait between the read(s) and the data being returned.

From this it appears that about 3 files per second is the most that can be achieved, even if only one read operation is needed for each file.

Correct.

I have now done a complete scan of my collection (just over 25,000 FLAC files) from the Pi. I did not do any precise measurements, but it seemed to be somewhat slower, closer to 2 files/sec.

That may be because the combination of the standard version of rclone (which is what I am using to effect a local mount of the OneDrive storage) and the LINUX filesystem does not offer the ability to use the two parameters that I was able to use on MacOS, namely (a) disabling the updating of file access time; and (b) specifying r/w size.

So it's possible that, with a bit of faffing about, I could improve the performance -- but, having now done the scan, I've no real reason to do so.

(30-11-2023 11:07)simoncn Wrote:  

(29-11-2023 21:19)GrumpyPatzer Wrote:  3. For each file, there are 16 individual read operations of 32kB each, so 512kB is being returned. (Based on what you say, it seems pretty likely that MinimServer is not asking for that much data. And the ‘head’ command only needs 32kB.)

Are the 16 reads just for a MinimServer scan or does this also happen for the 32 kB HEAD request?

It happened for both. Which reinforces my view that this may well be file system behaviour.

(30-11-2023 11:07)simoncn Wrote:  

(29-11-2023 21:19)GrumpyPatzer Wrote:  So it seems that — for both the ten ‘head’ commands and the MinimServer scan — a minimum of 512kB is being obtained, even though this *may* be more than is being requested.

I do not know why this should be the case. It is, I suppose, possible that this is Mac OS file handling behaviour. The fact that 512kB is still being obtained when I reduce the rclone parameters to 64kB suggests (although not conclusively) that the 512kB is not as a result of rclone behaviour. Also, if we accept that 512kB is being requested, it seems plausible (and is consistent with what I am observing) that reducing the rclone read-chunk size to 64kB would slow things down — if that is causing there to be more round trips to OneDrive.

This seems to be the most likely explanation of what you are seeing.

Quote:It may simply be that the latency of opening and obtaining the first chunk of data from a file stored in OneDrive is typically going to average around 300ms, at best, irrespective of broadband speed (as it happens, our broadband service was upgraded from 150 to 500Mbps today).

My broadband connection is about 70 Mbps and I see latency of a few ms for simple network requests. This suggests the 300 ms latency you are seeing is almost all caused by OneDrive and increasing network bandwidth will make very little difference to the overall latency.

Exactly. It's quite possible that there is no real mileage in spending too much more time on this -- particularly give that, for me at least, it's working quite satisfactorily.

(30-11-2023 11:07)simoncn Wrote:  

(29-11-2023 21:19)GrumpyPatzer Wrote:  Unless I can think of (or you can suggest) something else to try, I’ll probably leave it there, at least for now. When I get time to set up my new Pi (which has just arrived), I’ll set it up with MinimServer and rclone and will try playing music stored in OneDrive over a longer period of time. I suppose it’s possible that, if the 512kB is indeed a Mac OS characteristic, we may observe different behaviour with Raspbian or Ubuntu on a Pi.

I will be interested to hear whether this changes what you are seeing.

So, as noted above, the scan performance on the Pi seems somewhat worse, but it's really not a big problem. It seems to me rather nice to be able to play music from my collection held on cloud storage using something as simply as a Pi, with no NAS fan or spinning disk involved!

Andrew