Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Limit of 65000 folders
10-01-2013, 22:54 (This post was last modified: 10-01-2013 22:54 by Alekz.)
Post: #1
Limit of 65000 folders
Simon, what is the best workaround for more than 65k folders? Two instances of Minim?
Find all posts by this user
Quote this message in a reply
10-01-2013, 23:14
Post: #2
RE: Limit of 65000 folders
(10-01-2013 22:54)Alekz Wrote:  Simon, what is the best workaround for more than 65k folders? Two instances of Minim?

Do all these folders contain audio files? MinimServer ignores folders that don't contain any audio files, but it's possible that these folders might be counted as part of the 65000 limit. If this is the problem, I might be able to fix it.
Find all posts by this user
Quote this message in a reply
11-01-2013, 00:17
Post: #3
RE: Limit of 65000 folders
Under audio I have:

[/share/audio] # find . -type d | wc -l
8083

not only directories with audio files, but also with scans and videos. I just hit this limitation after I added several albums and restarted the server.
Find all posts by this user
Quote this message in a reply
11-01-2013, 01:30 (This post was last modified: 11-01-2013 01:30 by Alekz.)
Post: #4
RE: Limit of 65000 folders
Arrrrghhh... I know why.... There are two symbolic links pointing back to the same directory (for compatibility issues). find ignores links, Minim does not.
Find all posts by this user
Quote this message in a reply
11-01-2013, 07:13
Post: #5
RE: Limit of 65000 folders
(11-01-2013 01:30)Alekz Wrote:  Arrrrghhh... I know why.... There are two symbolic links pointing back to the same directory (for compatibility issues). find ignores links, Minim does not.

I was surprised that you could have a large enough music collection to occupy more than 65000 folders. Smile

Symbolic links can cause problems. As well as this, they can result in duplicate items in the MinimServer library if a link points to a directory that's underneath one of the top-level content directories.
Find all posts by this user
Quote this message in a reply
12-01-2013, 17:58
Post: #6
RE: Limit of 65000 folders
Well 65000 folders??? for what?? Do you have 65000 Albums Big Grin
Find all posts by this user
Quote this message in a reply
12-01-2013, 23:21 (This post was last modified: 12-01-2013 23:22 by Alekz.)
Post: #7
RE: Limit of 65000 folders
(12-01-2013 17:58)bbrip Wrote:  Well 65000 folders??? for what?? Do you have 65000 Albums Big Grin

See post #3 and #4
Find all posts by this user
Quote this message in a reply
04-09-2019, 22:08 (This post was last modified: 04-09-2019 22:44 by AndrewS.)
Post: #8
RE: Limit of 65000 folders
Hi Simon,

Posted about this yesterday, and you replied, but post no longer appears... (? forum was throwing 403 errors last night) So, carrying on the discussion...

Firstly, yes, all folders contain music files. I used to have a music store, and I've been curating my library a long time. 30,300 mp3 albums alone, which iTunes orders in artist then album folders = 47,000. Then I have Hi-res flac versions ripped of many of them. So it does add up.

OK, so running multiple instances of Minimserver on a Mac is not possible. That negates a possible solution (but does mean I don't have to puzzle out how to do it! Smile)

What about running two Macs (I have an older laptop), each running an external HD indexed by minimserver using unique hostnames? Would that create any incompatibilities (need to set different ports perhaps)? It'd mean splitting my library across drives, but that'd be easy enough.

May I ask whether a significant increase in the limit is possible? And I imagine there is a pragmatic reason (memory usage perhaps?) why the limit exists?

Any other suggestions?

Thank you
Find all posts by this user
Quote this message in a reply
05-09-2019, 08:17 (This post was last modified: 05-09-2019 08:17 by simoncn.)
Post: #9
RE: Limit of 65000 folders
Apologies for the forum issues. My hosting provider updated PHP to a newer version and this caused forum searches to stop working. I tried to fix this by updating to a later version of the forum software but something went wrong with the update and the forum was left in a semi-working state. After trying unsuccessfully to find a quick fix for this, I decided that the best way forward was to revert to a recent backup of the forum and redo the update using a different procedure. While all this was happening, I needed to take the forum offline, hence the 403 errors.

The 65000 folder limit is based on memory resource issues. When MinimServer was first released in 2012, a key feature was its ability to run on low-end devices, including some NAS devices with only 128MB of RAM for the operating system and all applications. As MinimServer holds key data structures in RAM for good performance, these structures needed to be as compact as possible and (without going into too much technical detail) this led to the 65000 folder limit. As the design point was to handle a library of 100,000 files (not folders), this number seemed sufficient and indeed has been sufficient for all users until now.

Things have moved on since then, with all current NAS devices having at least 256MB of RAM and some MinimServer users having very large libraries, so I will look again at how I can redesign the internal structures of MinimServer to remove the 65000 folder limit. One option is to allow up to 65000 folders for each content directory. As MinimServer can have up to 8 content directories, this would increase the limit to over 500,000 folders. Another option (which would consume more memory for small and medium-sized libraries) is to change the internal data structres of MinimServer to allow an unlimited number of folders.

This change will not happen immediately, so you need a workaround for now. Your suggestion of using a second Mac with a different hostname would work well. With this setup, there is no need to change port numbers because the MinimServer instances are differentiated by hostname, unlike the situation where multiple instances are running on the same machine.
Find all posts by this user
Quote this message in a reply
05-09-2019, 08:57
Post: #10
RE: Limit of 65000 folders
Hi Simon,

Thank you for your considered reply. I suspected that you'd had some issue requiring a restore from a recent 24hr backup. I hope you've got it all working satisfactorily now.

I am super excited that you are considering an update that will raise or remove the folder limit. Even the 8 content directory / 500,000 folder modification would work fine (for me anyway).

In the meantime, I can fire up the old laptop as a temporary work around. But having my entire library accessible from a single, unified index would of course be greatly preferable.

I appreciate that in the midst of your forum issue, you've taken my query into consideration and begun thinking of a solution. I hope it is straightforward to implement. If you need a beta tester, I have a suitable folder set to try it on! Smile

Best wishes,
A
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)