Normalize Unicode characters

[winstliu]

Requirements

• Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
• All new code requires tests to ensure against regressions

Description of the Change

Background: different Unicode sequences can be regarded as equivalent. On Windows and Linux, filenames are converted to NFC, or where characters like ñ are coded as one character. On macOS, filenames are converted to NFD, where ñ is coded as n + ◌̃. Both forms are considered to refer to the same character.

Fuzzy Finder was not taking Unicode normalization into account when searching for files. With this PR, NFD normalization for the filter query is now performed on macOS, while NFC normalization is performed for other platforms. This should allow for proper searching of filenames containing unicode characters.

Alternate Designs

NFKD and NFKC could be used instead, which would mean in addition the above, ff would also be treated as identical to ff.

Benefits

Fixes edge cases when searching for filenames with unicode characters.

Possible Drawbacks

None.

Applicable Issues

Fixes #69.

/cc @gjtorikian I think this should fix the issue you're seeing. If you'd like to test this PR but don't know how, I can provide guidance.

[winstliu]

Test plan:

1. Create a new file called erstiebegrüßung.html.
2. Open Fuzzy Finder using the fuzzy-finder:toggle-file-finder command.

macOS:

• Paste in erstiebegrüßung. The newly-added file should appear.
• Paste in erstiebegrüßung. The newly-added file should appear.

Windows:

• Paste in erstiebegrüßung. The newly-added file should appear.
• Paste in erstiebegrüßung. The newly-added file should appear.

Linux:

• Paste in erstiebegrüßung. The newly-added file should appear.
• Paste in erstiebegrüßung. The newly-added file should appear.

[lee-dohm]

Testing on macOS the first test case passes, the second one fails 😞

[lee-dohm]

Hmmmmm ... now I can't get either one to pass 😡

[rsese]

Had a chance to test on Linux (Ubuntu 16.04) - off of this branch, both tests passed for me (off of fuzzy-finder master, the first test passed and the second test failed).

[lee-dohm]

Tested this with the help of @50Wliu. Confirmed on Mac OS X 10.13.3 using the APFS file system, both tests fail. According to our research, APFS does not do Unicode file name normalization at all.

[lee-dohm]

@gjtorikian Can you confirm which OS version and filesystem you tested this on?

[lee-dohm]

Another helpful article on APFS file system normalization.

[gjtorikian]

I'm also on 10.13.3 and APFS.

[ghost]

    // Normalize to backslashes on Windows	
    if (process.platform === 'win32') {	

[ghost]

    // Normalize to backslashes on Windows	
    if (process.platform === 'win32') {	

Commit 05b0b51
#340: Normalize Unicode characters

[ghost]

Paste in erstiebegrüßung. The newly-added file should appear.
Paste in erstiebegrüßung. The newly-added file should appear.