Illegal Characters on Various Operating Systems

In the realm of cross-platform file sharing, sometimes the end users are not aware of the limitations posed by the operating system. This article will explain which characters can cause problems with files that use these characters.

Windows Conventions

The Windows operating system can use two different file systems, Protected-Mode File Allocation Table (FAT) file system and the New Technology File System (NTFS). The two systems have much in common, but the characters permitted in a file or folder name may differ. In the conventions listed below, it is true for both systems unless otherwise specified. Specifically there are cases where NTFS does not have the limitations (see note below).

The following characters are invalid as file or folder names on Windows using NTFS:

• / ? < > \ : * | "
• and any character you can type with the Ctrl key.
• In addition to the above illegal characters the caret ^ is also not permitted under Windows Operating Systems using the FAT file system.
• Under Windows using the FAT file system file and folder names may be up to 255 characters long.
• Under Windows using the NTFS file system file and folder names may be up to 256 characters long.
• Under Windows the length of a full path under both systems is 260 characters.
• In addition to these characters, the following conventions are also illegal:
• Placing a space at the end of the name.
• Placing a period at the end of the name.

The following file names are also reserved under Windows:

• com1
• com2
• com3
• com4
• com5
• com6
• com7
• com8
• com9
• lpt1
• lpt2
• lpt3
• lpt4
• lpt5
• lpt6
• lpt7
• lpt8
• lpt9
• con
• nul
• prn

Note: The previous conventions are true only if the application used in managing them does not use the Unicode API. Although the file system may support most of the above mentioned conventions the operating system may not. For example the NTFS file system allow paths to have a length up to 32,767 characters with each component (folder, file, etc.) being limited to 255 characters. However some windows applications like Explorer, for example, may not behave correctly in this circumstance. Other software, like ExtremeZ-IP uses the Unicode API so that file and folder names with invalid characters may be stored onto the NTFS file system.

Below the Macintosh conventions will touch upon the fact that the colon “:” is an invalid character in the Macintosh Operating Systems. Under Windows and the NTFS file system the colon is an illegal character, because it is used to open alternate file streams. However all other characters can be moved on and off the NTFS file system if a program with Unicode support is used. Both ExtremeZ-IP and MassTransit support this Unicode filenaming convention.

Macintosh OS9 Conventions

• The only illegal character for file and folder names in Mac OS 9 is the colon “:”
• File and folder names may be up to 31 characters in length.

Macintosh OS X Conventions

Since Mac OS X is build on top of UNIX there are a few inherent conventions that OS 9 users may not expect. Because of this, migrating certain files and folders from OS 9 to OS X may cause unexpected behavior.

• The only illegal character for file and folder names in Mac OS X is the colon “:”
• File and folder names are not permitted to begin with a dot “.”
• File and folder names may be up to 255 characters in length.

Invalid Characters Replacement Settings

The following setting is located in the MassTransit.cfg file:

%% MAP_INVALID_FILESYSTEM_CHARACTERS_AS_UNICODE = TRUE

If this value is false, MT will convert Unicode characters to a dash so filenames can be interpreted more clearly on Windows. Unicode characters do not appear in the filenames on Windows.
Set this value to true to disable replacing Unicode characters with '-'. Unicode characters will be preserved for files on Windows.

Note: When you make changes to the MassTransit configuration files you need to restart the MassTranist services for the changes to take effect. The correct way to do this operation is to stop the MassTransit Engine service, restart the MassTransit Transporter and start the MassTransit Engine again.

MassTransit Unicode Support

MassTransit 6.0.2 and later support transfers of files and folders with Unicode characters in TCP/IP, TCP/IP Secure, and UDT file transfers. By default, MassTransit will queue and transfer files and folders with high Unicode (>0xFF) successfully across the MassTransit protocol (TCP/IP, TCP/IP Secure, and UDT connections).

Note: Note that Unicode is not fully supported for web client HTTP transfers, hot folder transfers, FTP transfers, or with AppleScript. If you have Unicode support enabled and use Unicode filenames with these features, you may encounter file not found errors or munged filenames.

Note: Having Unicode support enabled on your server will still fail to transfer Unicode characters when connecting to legacy servers or servers with Unicode support disabled.

Polling and sending of Unicode files and folders is enabled by default, but can be disabled by editing the MassTransit.cfg file in the MassTransit application folder which by default is located in:

1. for MassTransit 7.6 and later:
   • on 64-bit machines:

   C:\Program Files (x86)\Acronis\MassTransit Server

2. for MassTransit 7.1 to 7.6:
   • on 32-bit machines:

   C:\Program Files\Group Logic\MassTransit Server

   • on 64-bit machines:

   C:\Program Files (x86)\Group Logic\MassTransit Server

3. for MassTransit 7.0.x:
   • on 32-bit machines:

   C:\Program Files\Group Logic\MassTransit Server 7

   • on 64-bit machines:
   • C:\Program Files (x86)\Group Logic\MassTransit Server 7

To disable unicode support:

1. Open the MassTransit.cfg file in an application suitable for plain text editing.
2. To prevent MassTransit from queuing files and folders with high Unicode characters, uncomment the following setting (delete the '%%' preceding the setting) and set it to FALSE as shown below:
            ENABLE_UNICODE_SUPPORT = FALSE
3. Save and close the MassTransit.cfg file.
4. Restart the MassTransit services in order to apply the changes immediately:
   1. Go to Start → Control Panel → Administrative Tools → Services;
   2. Highlight the MassTransit (for MassTransit SFTP, the service name is MassTransit SFTP) service and click on the Stop button from the Services tool bar, or select the Stop option from the context menu of the service.
   3. Highlight the MassTransit Transporter service and click on the Restart button from the Services tool bar, or select the Restart option from the context menu of the service;
   4. Highlight the MassTransit (for MassTransit SFTP, the service name is MassTransit SFTP) service and click on the Start button from the Services tool bar, or select the Start option from the context menu of the service.

There are a number of limitations to Unicode support:

• FTP and Hot Folder communication methods do not fully support Unicode characters. Most Unicode characters will be munged when transferred with these communication methods.
• Unicode is not supported for web client HTTP transfers. Both Windows and Mac web clients will present an error if a user attempts to upload files or folders containing Unicode characters. This limitation will be removed in a future release.
• Unicode characters in folder names will be replaced with dashes in web client downloads. This limitation will be removed in a future release.
• Unicode is not fully supported in the MassTransit user interface or email notifications. Unicode characters may not be properly displayed in the MassTransit Log, Files window, and email notifications containing file lists.
• MassTransit’s AppleScript interface does not support retrieval of Unicode characters. If a script retrieves a Unicode file name or folder name via the AppleScript interface, the file name will not be correct. This may impact the success of purging and workflow scripts processing files and folders with Unicode characters in the names.
• Programmatic submission of files containing Unicode characters (via COM, AppleScript, or SOAP) is not supported at this time.
• Only Unicode characters up to 0xFFFF are supported.

Note: It is highly recommended that you use the default settings for the existing Unicode flags – Replace Invalid Characters in File Names general privilege should be set to off for all contacts if Unicode is enabled and MAP_INVALID_FILESYSTEM_CHARACTERS_AS_UNICODE should be set to TRUE (or commented out) in the MassTransit.cfg file.

Examples of Unexpected Behavior

Below are a few scenarios that show what can happen if file names that are acceptable on one operating system are moved to another operating system:

Example 1:

1. Create a file named com1 on Mac OS 9.
2. Move the file to a Windows machine.
3. Under Windows 2000 viewing the folder which contains the file via Explorer will result in Explorer crashing.
4. Under Windows 2003 the file name cannot be changed because the file will require inherent access permissions.

Example 2:

1. Create a file named .text on Windows.
2. Move the file to a Mac OS X machine.
3. The file will not be visible via the Finder. (File and folder names beginning with a dot mean the file or folder is hidden)

Example 3:

1. Create a file named foo/ on Mac OS X.
2. Move the file to a Windows machine.
3. If the file is viewed via Explorer the file name will not appear as it did on the OS X machine.

Note: For more information on file naming refer to the following site: Microsoft Naming Conventions