Missing files --> Error: couldn't access file (with special characters) [SOLVED]

[blinkeye]

I'm using mp3tag with the following tag-to-filename conversion settings:

Code:

%artist% - %composer% - %album% - $num(%track%,2) - %title%

This results in filenames with special chararacters which cannot be parsed by minimserver:

Code:

Error: couldn't access file Chicago Symphony Orchestra - Sergey Rachmaninov - Rachmaninov Rhapsody on a Theme of Paganini - 27 - Noches en los jardines de Espa�a I. En el Generalife.flac

After logging in into the QNAP NAS I think to have the proper locale settings:

Code:

# locale -a
C
en_US.utf8
POSIX

and the filenames (even with special chars) are properly displayed:

Code:

# ls -l
-rw-r--r--    1 1004     1015     100574204 Feb 15 11:57 Chicago Symphony Orchestra - Sergey Rachmaninov - Rachmaninov Rhapsody on a Theme of Paganini - 27 - Noches en los jardines de España I. En el Generalife.flac

Is there anything I can do besides renaming all the files?

[blinkeye]

This also happens with German Umlaute characters:

Code:

Error: couldn't access file  Chicago Symphony Orchestra - Richard Wagner - Reiner Conducts Wagner - 02 - Die Meistersinger von N�rnberg, opera, WWV 96 Act III Prelude.flac

[simoncn]

(15-02-2015 12:29)blinkeye Wrote:  This also happens with German Umlaute characters:

Code:

Error: couldn't access file  Chicago Symphony Orchestra - Richard Wagner - Reiner Conducts Wagner - 02 - Die Meistersinger von N�rnberg, opera, WWV 96 Act III Prelude.flac

What output do you get from "locale" (not "locale -a")?

What is the platform default charset displayed in the MinimServer log?

Do you have 0.8.3 installed?

[blinkeye]

(15-02-2015 15:59)simoncn Wrote:  What output do you get from "locale" (not "locale -a")?

Code:

# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

(15-02-2015 15:59)simoncn Wrote:  What is the platform default charset displayed in the MinimServer log?

Do you have 0.8.3 installed?

Code:

# head -n 5 /share/Public/minimserver.log
MinimServer 0.8.3 update 57, Copyright (c) 2012-2014 Simon Nash. All rights reserved.
MinimStreamer 0.5.1, Copyright (c) 2012-2015 Simon Nash. All rights reserved.
Java(TM) SE Embedded Runtime Environment (build 1.8.0_06-b23)
Java HotSpot(TM) Embedded Client VM (build 25.6-b23, mixed mode)
Platform default charset is UTF-8
Language setting is 'auto'
starting MinimServer

[simoncn]

It looks like MinimServer is working correctly. This would mean that the problematic filenames don't contain UTF-8 characters but something else.

I presume you created these files using Mp3tag on Windows. If your Windows charset is Windows-1252, you might have filenames containing Windows-1252 characters instead of UTF-8 characters. How did you copy these files from Windows to the QNAP? If you browse these files using QNAP File Station, do the filenames look correct or incorrect?

[blinkeye]

You're right, it's a filename encoding issue:

Code:

$ isutf8 Chicago\ Symphony\ Orchestra\ -\ Sergey\ Rachmaninov\ -\ Rachmaninov\ Rhapsody\ on\ a\ Theme\ of\ Paganini\ -\ 27\ -\ Noches\ en\ los\ jardines\ de\ España\ I.\ En\ el\ Generalife.flac
Chicago Symphony Orchestra - Sergey Rachmaninov - Rachmaninov Rhapsody on a Theme of Paganini - 27 - Noches en los jardines de España I. En el Generalife.flac: line 2, char 1, byte offset 1: invalid UTF-8 code

I'm using mp3tag on Linux with Wine, but the file encoding seems to be latin1. What I now did was using convmv to convert files from iso-8859-1 to utf8:

Code:

convmv --notest -f iso-8859-1 -t utf8 file-with-encoding-issues.flac

As always, thank you Simon for the prompt and valid replies!

[MikeL]

Funny how problems multiply! I downloaded a couple of new flacs and could not understand why they were not indexing. Investigating led here: then I discovered that there were more than a hundred tracks failing that I had never noticed were missing. I think that the problem in my case leads back to unpredictable characters coming from cddb which I use to make the filenames.

Anyway, I wrote a little script to parse the log file and make the fixes. Here it is if anyone else wants it. It basically wraps convmv and uchardet (widely available).

Code:

#!/bin/bash
# uncomment or define on command line to make it happen
#DOIT="--notest"
LOG=/mnt/terabyte/music/utils/minimserver/data/minimserver.log

IFS=$'\n'

# convert all file names to UTF8 that appear as an error in log
#   File name as it appears in log is altered by error char
#     so make a stub /a/path/to/nn-* then get the real name
#     by eval'ing an ls on it.
#     Assumes that all files start with /nn-

for  i in $(awk '/access file/{print $5 }' $LOG);do
    STUMP=$(echo $i|sed 's/\(\/[0-9]\+[-_]\).*/\1*/')
    FILE=$(eval "ls $STUMP")
    FROM=$(echo $FILE|uchardet)
    echo $FROM
    if [[ -f $FILE ]]; then
        convmv -f $FROM -t UTF8 $FILE $DOIT
    fi
done

Obviously the LOG path needs to match the target system. Either uncomment #DOIT in the script or provide it on the command line change talk into action

Code:

chmod 777 ./fixnames.sh
DOIT=--notest ./fixname.sh