Unicode normalization of filenames

[tougher]

event.dataTransfer.files[0].name are getting the name of the file directly from the filesystem. This can cause problems when uploading files with special chars in the filename.

In my case I get Designha%u030Andbog.pdf (escaped with the JS escape() function) from the drop callback function, but i get Designh%E5ndbog.pdf from the done callback which originates from the server. The latter is the most correct. See my post at http://stackoverflow.com/questions/10805711/javascript-string-comparison-fails

To solve this issue, unicode normalization is needed (pointed out by slevithan at stackoverflow). This could be done in the _normalizeFile function, but it will require a lot of code.

[blueimp]

Thanks for your post and pointing out the related discussion on StackOverflow.
Unfortunately, you have to do the unicode normalization on server-side, simply because the name attribute of File objects is read only.

Only some browsers support a third parameter for FormData.append, which could possibly be used to transfer a normalized version of the file name, but would require storing it as a separate, writable attribute of the File object:
https://github.com/blueimp/jQuery-File-Upload/blob/master/js/jquery.fileupload.js#L357

I'm not sure if I should incorporate unicode normalization into the example PHP upload handler class (or the other server versions), as there might be different opinions on the normalization forms used.
However, here's the relevant PHP class:
http://php.net/manual/en/class.normalizer.php

[tougher]

Damn, you are right :-)
My solution is to do the normalization in the callback functions. I'm using this lib https://github.com/walling/unorm.
I guess the normalization is done automatically in my server-side code (PHP), I haven't done anything to correct the strings.

Thanks for an awesome framework btw!

[pomartel]

But what if the upload is done directly to S3 ? The filename cannot be normalized on the server side and as @blueimp points out, the File object is read-only...

[blueimp]

jQuery File Upload will use the name property of the file and pass it to FormData.append as third argument.
This is not yet supported by Firefox, but Chrome will use this optional third parameter as filename for the file to upload (didn't test if it is supported by Opera or IE10).

You can use it for arbitrary files by applying the slice method to copy the complete files:

var blob = file.slice(0, file.size, file.type);
blob.name = normalizedFileName;

[pomartel]

I think the best way to go would be to not mess with the original encoding and keep the filename as it is but it seems that at some point, the precomposed characters are converted back to combined characters. That's what I'm explaining here : waynehoover/s3_direct_upload#19 (comment)

The response I get from S3 looks like this in Chrome :

<PostResponse>
  <Location>https://poll-development-uploads.s3.amazonaws.com/uploads%2F81c83441728844aab30bb85cc3f8a2a8%2Fch%C3%A9.jpg</Location>
  <Bucket>poll-development-uploads</Bucket>
  <Key>uploads/81c83441728844aab30bb85cc3f8a2a8/ché.jpg</Key>
  <ETag>"1e5bcc527888aa9ac045d6c98fc7db14"</ETag>
</PostResponse>

The encoding is still the same as the original one and if I take the URL from the Location tag and put it in my browser, it works. But when I read the filename in the done callback using data.files[0].name, it looks like the encoding has changed. This means the re-encoding happens in the Jquery File Upload library somewhere. But I cannot find any encode method being used anywhere and I'm not familiar with the framework.

Could there be any re-enconding happening with the S3 Post Response ? Where is this taking place ?

[blueimp]

data.files[0].name shouldn't change, since this is effectively the same variable in the add and done callbacks.
If you test the following code on the demo with files with arbitrary filenames you should always get true as the comparison result in the console:

var a,b;
$('#fileupload').on('fileuploadadd', function (e, data) {
    a = data.files[0].name;
});
$('#fileupload').on('fileuploaddone', function (e, data) {
    b = data.files[0].name;
    console.log(a, b, a === b);
});

If however you meant to compare data.files[0].name with data.result[0].name, which can be done with the following code, then yes, you'll get false in the demo as well for comparison results when you upload files with non-ASCII filenames:

var a,b;
$('#fileupload').on('fileuploadadd', function (e, data) {
    a = data.files[0].name;
});
$('#fileupload').on('fileuploaddone', function (e, data) {
    b = data.result[0].name;
    console.log(a, b, a === b);
});

However, data.result is a property filled with the JSON response from server-side.
In any case, the solution is to normalize the file names so they use the same Unicode representation.

[pomartel]

Thanks for your help @blueimp ! Turns out you were right, the filename S3 sends back is encoded differently than the one that is uploaded. I use the data.result to avoid encoding confusion instead of data.files[0].name. I have been testing it and it seems to fix the bug. Thanks again for your time!

[blueimp]

No worries and thanks for coming back and posting your results. :)

[cokron]

The unorm did the trick! Took me like three hours until I landed on this page for the second time and followed the advice. Thank you, guys!