The Hidden & Dangerous UTF encoding traps

Now let’s face this, we all work with Text in our day to day lives [If you don’t, kindly do let me know what kind of work you do?] and it is funny how we ignore some basics of the game. I’ve been myself victim of these traps before but as I spent more time dealing with them I found out some really simple solutions to some really strange problems.

Recently, a friend of mine was testing an MSI Installer which was custom built by his Dev team for automated installation and deployment. One of the purposes of that installer was to auto configure certain server parameters. Now that configuration was done using a VBS. Now they were rushing for a release and timeline was short. As such it was a straight forward process, so he ran the MSI and within a few seconds he was presented with a weird error. And Installer aborted. Now this was not the first time that installer was being used. It was used across teams in that group and was in practice for a while now. I was standing by him when this was going on so I offered him my help and he happily agreed.

Now first thing I did is to check Eventlog. [I work with Microsoft Dynamics CRM most of the times, so this has become my habit] and indeed there was an error recorded under Windows Installer. On checking that error we found out a script necessary to run this setup was failing and due to that failure Setup cannot continue.  So we knew that it is coming from

So I asked him to run the script manually by double clicking it. The script instantly gave an error, and the error was there was an invalid character on line 1, position 0. So we opened the file in Visual Studio and it was a simple On Error Resume Next nothing fancy no magic there.

Now the same installer approach was working fine for other teams and my friend just copied the scripts and made his own installer. Now keyword here is “copied”, so I asked him from where and how did he copy it? He said he used Remote Desktop. And it striked me that this could be the issue because of the codepage and maybe during the copy something was messed up. I asked him to copy the script file to his notepad and save it as a VBS file and then putting it back. The same file and the same content worked absolutely fine.

Same goes with XML as well, the encoding type specified in header can baffle you sometimes. A perfectly valid XML might also give an error if you try to open it with Internet Explorer 8 or below.

So next time you are facing some weird error in otherwise perfect text file I suggest you try fix the encoding before you dive in somewhere else.

Leave a Reply

Your email address will not be published. Required fields are marked *