special characters in file/directorynames

[angelus1969]

I have seen that this issue was reported before, but that was for the NAS build. I'm using Minimserver on an openvz (Proxmox) Ubuntu box, 12.04 x64. File and or directorynames like for example Arvo Pärt won't get indexed unless I change it to Arvo Part.

[simoncn]

(17-12-2013 09:41)angelus1969 Wrote:  I have seen that this issue was reported before, but that was for the NAS build. I'm using Minimserver on an openvz (Proxmox) Ubuntu box, 12.04 x64. File and or directorynames like for example Arvo Pärt won't get indexed unless I change it to Arvo Part.

Your locale charset needs to be UTF-8 for this to work. What output do you get from the 'locale' comand?

[angelus1969]

(17-12-2013 10:43)simoncn Wrote:  

(17-12-2013 09:41)angelus1969 Wrote:  I have seen that this issue was reported before, but that was for the NAS build. I'm using Minimserver on an openvz (Proxmox) Ubuntu box, 12.04 x64. File and or directorynames like for example Arvo Pärt won't get indexed unless I change it to Arvo Part.

Your locale charset needs to be UTF-8 for this to work. What output do you get from the 'locale' comand?

It is set:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

[simoncn]

(17-12-2013 10:45)angelus1969 Wrote:  It is set:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

This looks OK. What charset does MinimServer report at the start of the log messages?

[angelus1969]

(17-12-2013 10:57)simoncn Wrote:  

(17-12-2013 10:45)angelus1969 Wrote:  It is set:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

This looks OK. What charset does MinimServer report at the start of the log messages?

Now it says: Platform default charset is US-ASCII but before it said: Platform default charset is UTF-8
Not sure how this got changed..... I just upgraded from 0.7.2 to 0.8.1 but as I scroll back in the terminal window, I can see that in the log when I first started 0.8.1 it also said UTF-8. After a reboot of the VZ it now says US-ASCII. Not sure what this means for the characters....

If I SSH into this VZ and go to a directory where there is an accented character, it shows without problem. When I use 'folder view' inside an UPNP controller, it doesn't show the same directory at all. This was the same with the 0.7.2 version.

Mind you, there is no problem if there are accented characters inside the tags, it is only a problem if the directory or filename have special characters.


---
Regards, Angelo

[simoncn]

(17-12-2013 11:01)angelus1969 Wrote:  If I SSH into this VZ and go to a directory where there is an accented character, it shows without problem. When I use 'folder view' inside an UPNP controller, it doesn't show the same directory at all. This was the same with the 0.7.2 version.

Mind you, there is no problem if there are accented characters inside the tags, it is only a problem if the directory or filename have special characters.

These symptoms are expected if the charset isn't set to UTF-8 when MinimServer is launched. Are you launching MinimServer manually using the startd command from a login shell, or have you set MinimServer to start automatically when Linux is booted? If it's the latter, how did you set up the automatic start?

[angelus1969]

(17-12-2013 17:25)simoncn Wrote:  

(17-12-2013 11:01)angelus1969 Wrote:  If I SSH into this VZ and go to a directory where there is an accented character, it shows without problem. When I use 'folder view' inside an UPNP controller, it doesn't show the same directory at all. This was the same with the 0.7.2 version.

Mind you, there is no problem if there are accented characters inside the tags, it is only a problem if the directory or filename have special characters.

These symptoms are expected if the charset isn't set to UTF-8 when MinimServer is launched. Are you launching MinimServer manually using the startd command from a login shell, or have you set MinimServer to start automatically when Linux is booted? If it's the latter, how did you set up the automatic start?

Now it is set up to start automatically. I have made an entry in the /etc/rc.local file.

Edit: you are right. When starting manually, it says 'utf8' in the log and a rescan shows the 'missing' folders and files..... How can I set it for autostart but with the correct setting for the locale? Creating a script inside /etc/init.d doesn't work properly either, the result is also US_ASCII.......


---
Regards, Angelo

[simoncn]

(17-12-2013 17:46)angelus1969 Wrote:  Now it is set up to start automatically. I have made an entry in the /etc/rc.local file.

Edit: you are right. When starting manually, it says 'utf8' in the log and a rescan shows the 'missing' folders and files..... How can I set it for autostart but with the correct setting for the locale? Creating a script inside /etc/init.d doesn't work properly either, the result is also US_ASCII.......

The charset problem is caused by starting MinimServer from /etc/rc.local. For some reason, this doesn't set the correct charset.

The minimserver/bin/setup script gives you the option to set up automatic startup for MinimServer. If you use this script to set up automatic startup, I believe you will get the correct charset.

I will investigate the problem that you reported with the minimserver/bin/setup script and I will post here when I have more information.

[simoncn]

(17-12-2013 19:07)simoncn Wrote:  I will investigate the problem that you reported with the minimserver/bin/setup script and I will post here when I have more information.

I am attaching a .tar.gz archive containing a slightly updated version of the minimserver/bin/setup script. Please extract the setup file from this archive and replace the version in /root/minimserver/minimserver-0.8.1/bin/setup with this version, then run the new script by typing /root/minimserver/bin/setup. If you still get a failure, please post the error message.

[angelus1969]

(17-12-2013 21:18)simoncn Wrote:  

(17-12-2013 19:07)simoncn Wrote:  I will investigate the problem that you reported with the minimserver/bin/setup script and I will post here when I have more information.

I am attaching a .tar.gz archive containing a slightly updated version of the minimserver/bin/setup script. Please extract the setup file from this archive and replace the version in /root/minimserver/minimserver-0.8.1/bin/setup with this version, then run the new script by typing /root/minimserver/bin/setup. If you still get a failure, please post the error message.

I did as you asked, but when I start the script from the ~/minimserver/bin directory it still states the same error. But that is to be expected, because the new script is in ~/minimserver/minimserver-0.8.1/bin directory....... Why are there 2 bin directories? I first thought one was links to the other, but that is not the case!
root@mminim:~/minimserver/minimserver-0.8.1/bin# ls -lha
total 36K
drwxr-xr-x 2 root root 4.0K Dec 17 23:53 .
drwxr-xr-x 5 root root 4.0K Dec 17 04:30 ..
-rwxr-xr-x 1 root root 7.8K Dec 17 23:50 setup
-rwxr-xr-x 1 root root 162 Dec 17 04:30 startc
-rwxr-xr-x 1 root root 192 Dec 17 04:30 startd
-rwxr-xr-x 1 root root 982 Dec 17 04:30 stopall
-rwxr-xr-x 1 root root 1.8K Dec 17 04:30 uninstall
-rwxr-xr-x 1 root root 2.6K Dec 17 04:30 update
root@mminim:~/minimserver/minimserver-0.8.1/bin# ls -lha ../../bin/
total 36K
drwxr-xr-x 2 root root 4.0K Dec 17 04:30 .
drwxr-xr-x 9 root root 4.0K Dec 17 04:30 ..
-rwxr-xr-x 1 root root 7.8K Dec 17 04:30 setup
-rwxr-xr-x 1 root root 162 Dec 17 04:30 startc
-rwxr-xr-x 1 root root 192 Dec 17 04:30 startd
-rwxr-xr-x 1 root root 982 Dec 17 04:30 stopall
-rwxr-xr-x 1 root root 1.8K Dec 17 04:30 uninstall
-rwxr-xr-x 1 root root 2.6K Dec 17 04:30 update
root@mminim:~/minimserver/minimserver-0.8.1/bin#

EDIT: I have a second VZ where I also installed the new version, so I could test a little more without breaking my main VZ. I executed the new setup from the ~/minimserver/minimserver-0.8.1/bin directory and this script works. It creates an executable script inside /etc/init.d and creates the runlevel info. After a restart however, minimserver is not started. When I try to start the server by hand with root@mminim:/etc# init.d/minimserver start
it states: /root/minimserver/minimserver-0.8.1/bin/startd: 4: cd: can't cd to /root/minimserver/minimserver-0.8.1/bin/../data
EDIT #2: found it, by running the setup script from ~/minimserver/minimserver-0.8.1/bin causes the init.d script to read: su "root" -c "/root/minimserver/minimserver-0.8.1/bin/startd" while it should be su "root" -c "/root/minimserver/bin/startd"

EDIT #3: And alas, it doesn't help with the original problem. Minimserver now starts okay after a reboot, but the locale is still US_ASCII when starting it this way...... What I understand after some web-searching, is that this is a known problem and the script should explicit state the locale itself.... Not sure how (yet).....

EDIT#4: running the command update-locale LANG=<desired locale> creates a necessary file in /etc/default After a reboot minimserver starts with the correct locale.....


---
Regards, Angelo