Foreign/accented characters in Playlists

Support questions for Neutron Player only.
BerlinFiioFan
Posts: 5
Joined: Tue May 05, 2020 7:13 pm

Foreign/accented characters in Playlists

Post by BerlinFiioFan » Wed Sep 30, 2020 12:30 pm

Hello --

I have created some playlists via an external application, and when I import the playlist onto my FiiO device, songs that have accented characters (or in some cases, Japanese and Korean characters) are not imported.

For example:
#EXTINF:157, another silent weekend, afternoon bike ride - Ricefields in Nagoya (feat. Afternoon Bike Ride)
Music/JULAUGSEP20/another silent weekend/Pen Pal (Album)/08 Ricefields in Nagoya (feat. Afternoon Bike Ride).flac
#EXTINF:226, Hubert Kah - Rosemarie
Music/C/Compilations/Total Phänomenal- Neue Deutsche Welle/01-03 Rosemarie.Mp3

The first track imports, but the 2nd one does not. I noticed that all the tracks that did not import have some sort of non-standard characters in them.

Is there some way to correct this?

Thanks!

blaubär
Posts: 3313
Joined: Tue Apr 02, 2019 6:48 am

Re: Foreign/accented characters in Playlists

Post by blaubär » Wed Sep 30, 2020 1:19 pm

You could compare the character set of the externally created playlists with the character set of playlists created by Neutron ( probably UTF-8 Unicode ) and then try to convert the externally created files into this character set, thereby replacing characters unknown in Neutron's character set with suitable other existing characters.

BerlinFiioFan
Posts: 5
Joined: Tue May 05, 2020 7:13 pm

Re: Foreign/accented characters in Playlists

Post by BerlinFiioFan » Wed Sep 30, 2020 1:44 pm

Excellent suggestion, thank you!

The external file shows the entry as this:
Music/C/Compilations/Total PhaÃànomenal- Neue Deutsche Welle/01-03 Rosemarie.Mp3

And Netron shows it like:
Music/C/Compilations/Total Phänomenal- Neue Deutsche Welle/01-03 Rosemarie.Mp3

Guess I have to see if the application I'm using to create the playlists (Swinsian) supports UTF-8 Unicode output.

Thanks again!

blaubär
Posts: 3313
Joined: Tue Apr 02, 2019 6:48 am

Re: Foreign/accented characters in Playlists

Post by blaubär » Wed Sep 30, 2020 2:42 pm

You're welcome. You're not the only user ( and Neutron isn't the only android player ) affected by this, see MinimServer Forum

WorLord
Posts: 3
Joined: Thu Mar 18, 2021 3:58 pm

BUG: Playlists dropping songs containing accent marks and umlauts (è, é, and ö)

Post by WorLord » Thu Mar 18, 2021 4:18 pm

Greetings!

I have a music collection of some 14,000 songs that I've been meticulously curating for years. I have a set of m3u8 playlists that go with it, one in particular of every favored song from every album I own.

Yesterday I noticed that while most of the songs in the imported playlist are there (3441), almost a hundred were missing (Neutron somehow trimmed it to 3368).

After quite a lot of troubleshooting, I found out that, to a track, each of the songs Neutron "dropped" from the playlist contained special characters in either the track, album, or artist name; most often the ones in the title (è, é, and ö), but not limited to those. Curiously, some special characters were not similarly dropped (Æ, ², and the entire album name "Полная шляпа" seem to be just fine).

What's really curious is that all of the songs and albums referenced by these playlists are indeed recognized by Neutron. I'm not _missing_ any music: I can go to the artist just fine and play the very songs referenced. It's just that when I change and re-import the playlist, those songs get dropped from it when I first open it.

I have corrected this in the metadata itself by dropping those special characters from the artist, album, and song names -- for example, changing "Queensrÿche" to "Queensryche" and re-uploading all those tracks and updated playlist. But that was an arduous task and ultimately I'd like to see the bug fixed.

Please let me know if there is any logging or diagnostic data I can get to help aid in this task. Neutron lives on a Samsung Galaxy S8 Active; the music collection is stored on an SD Card formatted with Fat32 (if that information helps any).

blaubär
Posts: 3313
Joined: Tue Apr 02, 2019 6:48 am

Re: Foreign/accented characters in Playlists

Post by blaubär » Thu Mar 18, 2021 5:06 pm

WorLord wrote:
Thu Mar 18, 2021 4:18 pm
After quite a lot of troubleshooting, I found out that, to a track, each of the songs Neutron "dropped" from the playlist contained special characters in either the track, album, or artist name; most often the ones in the title (è, é, and ö), but not limited to those. Curiously, some special characters were not similarly dropped (Æ, ², and the entire album name "Полная шляпа" seem to be just fine).

What's really curious is that all of the songs and albums referenced by these playlists are indeed recognized by Neutron. I'm not _missing_ any music: I can go to the artist just fine and play the very songs referenced. It's just that when I change and re-import the playlist, those songs get dropped from it when I first open it.
I've merged the topics. Perhaps the solution suggested there could also help in your case.
blaubär wrote:
Wed Sep 30, 2020 1:19 pm
You could compare the character set of the externally created playlists with the character set of playlists created by Neutron ( probably UTF-8 Unicode ) and then try to convert the externally created files into this character set, thereby replacing characters unknown in Neutron's character set with suitable other existing characters.
BerlinFiioFan wrote:
Wed Sep 30, 2020 1:44 pm
The external file shows the entry as this:
Music/C/Compilations/Total PhaÃànomenal- Neue Deutsche Welle/01-03 Rosemarie.Mp3

And Netron shows it like:
Music/C/Compilations/Total Phänomenal- Neue Deutsche Welle/01-03 Rosemarie.Mp3

Guess I have to see if the application I'm using to create the playlists (Swinsian) supports UTF-8 Unicode output.

WorLord
Posts: 3
Joined: Thu Mar 18, 2021 3:58 pm

Re: Foreign/accented characters in Playlists

Post by WorLord » Thu Mar 18, 2021 6:29 pm

Unfortunately, it does not. I've very thoroughly examined the files I generate -- m3u8's, all (the "8" specifies UTF-8, and my editor can confirm it's UTF-8), and the characters in the file appear exactly as they do in the filename. Additionally, some UTF-8 characters _do_ survive, as highlighted in the original post.

Are there any other ideas?

blaubär
Posts: 3313
Joined: Tue Apr 02, 2019 6:48 am

Re: Foreign/accented characters in Playlists

Post by blaubär » Thu Mar 18, 2021 8:40 pm

Okay, so the files you generate have the same character set as the playlist files generated by Neutron ? I suggest you contact neutronmp@gmail.com and send them an example file.

WorLord
Posts: 3
Joined: Thu Mar 18, 2021 3:58 pm

Re: Foreign/accented characters in Playlists

Post by WorLord » Fri Mar 19, 2021 3:50 am

Just wanted to follow up: I don't think this is Neutron's problem.

My own words actually gave away the culprit: SDCard was formatted as Fat32. I'm... not sure what compatibility Fat32 has with UTF-8, but it is apparently not the best. Rumor on the street is that Android phones have been formatting SDCards as exFAT for a while now, so I decided to nuke the card from orbit and let the phone format it.

After transferring all my files and playlists over, everything shows up. I've even changed some songs back to contain the umlauts and what have you, and re-synced (heavens above praise Syncthing)... and my songs are still fine.

I'm sorry to have bothered y'all, but hopefully someone will see this and know what to do.

blaubär
Posts: 3313
Joined: Tue Apr 02, 2019 6:48 am

Re: Foreign/accented characters in Playlists

Post by blaubär » Fri Mar 19, 2021 4:42 am

WorLord wrote:
Fri Mar 19, 2021 3:50 am
I'm... not sure what compatibility Fat32 has with UTF-8, but it is apparently not the best.
Thank you for the feedback ! Yes, it would seem that FAT32 doesn't use UTF for the filenames.

Unicode filenames on FAT-32?
[...]
But official MSDN documentation is very vague regarding what codepage(s) is used to store filenames (filepaths) on FAT-32.
[...]
Here it says that OEM code page (CP437 I assume) is used to store filenames:
[...]
I don't believe FAT is directly capable of either UTF-16 or UTF-8.
[...]
On a typical American system, the OEM code page is "CP437", but the Windows code page is Windows-1252
[...]
NTFS vs FAT vs exFAT
File System Features / Unicode File Names
[...]
[...] exFAT / Unicode Character Set
[...] FAT32 / System Character Set
[...]

Post Reply

Who is online

Users browsing this forum: No registered users and 50 guests