Streaming music from cloud storage

[GrumpyPatzer]

Thank you for another comprehensive and clear reply. That all makes good sense.

As I mentioned, the best scan performance I have managed to obtain (no more than around 3 files / sec) is with counter-intuitively large FUSE-T and rclone parameters (e.g. 512kB for rclone read chunk and buffer size), even though not that much data is being read. It is mildly irritating not to be able to work out why this is happening (although that is in no way an implied criticism of MinimServer; I strongly suspect that this is happening because of the way Mac OS, rclone and FUSE-T interact).

So, at the risk of spending too much time on this (given that 3 files / sec is probably adequate for an operation that, in effect, only needs to be done once), I have done two things: used WireShark to capture what is happening on the lo0 interface; and a simple test (on the command line) to obtain the first 32kB from files stored on OneDrive. The way I did the command line test was simply to loop round doing a ‘head -c 32768 …’ command for ten FLAC files on OneDrive. (It needed to be ten different files, because once a file has been read, a second read will be performed *much* more quickly. It is clearly being cached or buffered, probably by rclone, even though I have got VFS caching disabled.)

Using WireShark, I can see the NFS open operation for each file, a set of read operations and the data coming back from OneDrive. Main observations:

1. In both cases (command line test and MinimServer scan), it is typically taking between 200 and 350ms for each file operation, usually over 250ms but occasionally spiking to over 500ms. Unsurprisingly, the bulk of that time is that wait between the read(s) and the data being returned.

2. I can see the FLAC tag information in the first 32k chunk of data, so am fairly confident that I am observing the right things.

3. For each file, there are 16 individual read operations of 32kB each, so 512kB is being returned. (Based on what you say, it seems pretty likely that MinimServer is not asking for that much data. And the ‘head’ command only needs 32kB.)

4. If I reduce the rclone buffer and read-chunk parameters to 64k, the whole operation (to open the file and return 512kB) takes several times longer, typically well over 1s.

So it seems that — for both the ten ‘head’ commands and the MinimServer scan — a minimum of 512kB is being obtained, even though this *may* be more than is being requested.

I do not know why this should be the case. It is, I suppose, possible that this is Mac OS file handling behaviour. The fact that 512kB is still being obtained when I reduce the rclone parameters to 64kB suggests (although not conclusively) that the 512kB is not as a result of rclone behaviour. Also, if we accept that 512kB is being requested, it seems plausible (and is consistent with what I am observing) that reducing the rclone read-chunk size to 64kB would slow things down — if that is causing there to be more round trips to OneDrive.

It may simply be that the latency of opening and obtaining the first chunk of data from a file stored in OneDrive is typically going to average around 300ms, at best, irrespective of broadband speed (as it happens, our broadband service was upgraded from 150 to 500Mbps today).

Unless I can think of (or you can suggest) something else to try, I’ll probably leave it there, at least for now. When I get time to set up my new Pi (which has just arrived), I’ll set it up with MinimServer and rclone and will try playing music stored in OneDrive over a longer period of time. I suppose it’s possible that, if the 512kB is indeed a Mac OS characteristic, we may observe different behaviour with Raspbian or Ubuntu on a Pi.

Andrew