®
IBM Software Group
WebSphere MQ Data Conversion
Blanche Ishida and Karen Stanley
WebSphere MQ Support
WebSphere® Support Technical Exchange
IBM Software Group
Agenda
Terms and Definitions
WebSphere MQ Message Header (MQMD)
Customer Scenarios
IBM Technotes
Useful Links
Summary
Questions and Answers
WebSphere® Support Technical Exchange
2
IBM Software Group
Terms and Definitions
Character Set
Is a defined set of characters used to represent textual
information (e.g. 0-9, a-z, A-Z, .,;:!?/-)
A Character Set generally supports more than one
language
Code Page
(AKA Code set) A code page can be defined as a table
with a mapping of alphanumeric code and its binary
representation.
Where each character in a character set is assigned a
numerical representation (often used interchangeably
with character set – e.g. charset in HTML)
WebSphere® Support Technical Exchange
3
IBM Software Group
Terms and Definitions
A code point is the location of a character within
the code page
CCSID Coded Character Set ID is a unique
number (0-65535) used by IBM to uniquely identify
a Coded Character Set and a Codepage.
WebSphere® Support Technical Exchange
4
IBM Software Group
Terms and Definitions
ASCII : American Standard Code for Information Interchange.
The code used as the basis for most codes apart from
EBCDIC.
EBCDIC : Extended Binary Coded Decimal Information Code
(EBCDIC). It is typically used on z/OS, iSeries and VSE
UNICODE : A code page which defines most of the
characters used by the languages now used throughout the
world. UCS-2, UTF-8, UTF-16 and UTF-32 for encodings.
SBCS Single Byte Character Set. A code page with up to 256
characters which can be represented by a single 8 bit byte.
DBCS Double Byte Character Set. A code page with up to
65536 characters.
WebSphere® Support Technical Exchange
5
IBM Software Group
MQMD
The Message Descriptor is not entirely character data, so
some of it will be converted as character data, some as
numeric, and some is passed unconverted.
Note: MsgId, CorrelId, and Accounting Token are NOT
considered to be character data, and as such WILL NOT be
converted as character data. Many applications put
character data into these fields, but if the message is being
exchanged between ASCII and EBCDIC platforms, and the
application on the other side needs to be able to read these
fields, then it is the responsibility of the application (or a user
exit) to convert these fields. Also the data maybe in
hexadecimal representation for MsgID and CorrelID.
WebSphere® Support Technical Exchange
6
IBM Software Group
WebSphere MQ Message Header
StrucId Structure identifier
Version Structure version
number
Report Options for report
messages
MsgType Message type
Expiry Message lifetime
Feedback Feedback or reason
code
Encoding Numeric encoding of
message data
CodedCharSetId Character set
identifier of message data
Format Format name of
message data
WebSphere® Support Technical Exchange
Priority Message priority
Persistence Message persistence
MsgId Message identifier
CorrelId Correlation identifier
BackoutCount Backout counter
ReplyToQ Name of reply queue
ReplyToQMgr Name of reply queue
manager
UserIdentifier User identifier
AccountingToken Accounting token
7
IBM Software Group
WebSphere MQ Message Header cont…
ApplIdentityData Application
data relating to identity
PutApplType Type of
application that put the
message
PutApplName Name of
application that put the
message
PutDate Date when message
was put in GMT
PutTime Time when message
was put in GMT
ApplOriginData Application data
relating to origin
WebSphere® Support Technical Exchange
Note: The remaining fields are
ignored if Version is less than
MQMD_VERSION_2.
GroupId Group identifier
MsgSeqNumber Sequence number
of logical message within group
Offset Offset of data in physical
message from start of logical
message
MsgFlags Message flags
OriginalLength Length of original
message
8
IBM Software Group
MQMD
There are two parts to a MQ message.
There is a Message Descriptor (MQMD), followed by User data.
D4C44040 00000001 00000000 00000008 MD ............
00007530 00000000 00000311 000001F4
..............4
D4D8E2E3 D9404040 00000000 00000000 MQSTR ........
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 546 CodedCharSetId : 437
Format : 'MQSTR '
Priority : 0 Persistence : 0
WebSphere® Support Technical Exchange
9
IBM Software Group
MQMD : Encoding
Encoding is generally taken to mean the method
that this platform uses to represent numeric data.
There are two general types:
‘LittleEndian’, used by Intel processors (e.g
Windows, Linux on Intel). e.g. the number 437
would be represented (in hex) as X’BF01’ (we
refer to this as byte swap)
‘BigEndian’. e.g. the number 437 would be
represented (in hex) as X’01BF’.
WebSphere® Support Technical Exchange
10
IBM Software Group
Encoding scheme by platform
Platform Encoding
zSeries
AIX
iSeries
HP-UX
Windows
SINIX
Sun Solaris (on SPARC processors)
Sun Solaris (on INTEL processors)
Linux (Intel)
Linux (zSeries)
NonStop Kernel (NSK)
OVMS Alpha
Open VMS VAX
Tru64 Unix
WebSphere® Support Technical Exchange
Scheme
BigEndian
BigEndian
BigEndian
BigEndian
LittleEndian
BigEndian
BigEndian
LittleEndian
LittleEndian
BigEndian
BigEndian
BigEndian
BigEndian
BigEndian
11
IBM Software Group
CCSID
The Queue Manager CCSID is important as it
defines the default CCSID for messages being
created using MQPUT and is the default CCSID
which will be used for conversion when messages
are retrieved using MQGET. If the channel option
CONVERT(YES) is set the conversion is attempted
for the receiving Queue Manager’s CCSID.
To display the queue manager CCSID use the
DIS QMGR command
WebSphere® Support Technical Exchange
12
IBM Software Group
MQMD : CodedCharSetId
Some well known CCSID’s:
CCSID 437 -- this is an ASCII code page, used mainly under
OS/2, DOS, and Microsoft Windows console (OEM) windows.
CCSID 850 -- this is another common ASCII code page, used
mainly on under OS/2, DOS, and Microsoft Windows console
(OEM) windows used on PCs in Europe and some AIX
locales.
CCSID 819 -- this is the ISO 8859-1 standard Western
European ASCII code page. Used by most UNIX locales.
CCSID 500 -- This is an EBCDIC code page, used mainly on
z/OS. It is known as the ‘International’ codepage.
CCSID 037 -- This is another popular EBCDIC code page,
used on OS/400. It is the ‘US English’ codepage.
WebSphere® Support Technical Exchange
13
IBM Software Group
MQMD : Format
This indicates to the receiver of a message the format of the
application data in the message.
The queue manager has a number of built-in formats
with names beginning MQ, for example MQFMT_STRING. If these
do not meet your needs, you can define your own formats (userdefined formats), but you must not use names beginning with MQ for
these. When you create and use your own formats, you must write a
data-conversion exit to support a program getting the message
using MQGMO_CONVERT.
The format name MQFMT_NONE is a special value that indicates
that the nature of the data in the message is undefined. As a
consequence, the queue manager does not attempt conversion
when the message is retrieved from the queue.
WebSphere® Support Technical Exchange
14
IBM Software Group
FORMAT
The queue manager built-in formats all have names beginning with
MQFMT. They are listed and described in the WebSphere MQ
Application Programming Reference under the Format field of the
Message descriptor (MQMD)
Some built-in formats
MQFMT_NONE (default)
MQFMT_STRING (for character data conversion)
MQFMT_DEAD_LETTER_HEADER
MQFMT_IMS
MQFMT_IMS_VAR_STRING
MQFMT_PCF
MQFMT_RF_HEADER
MQFMT_RF_HEADER_2
MQFMT_TRIGGER
MQFMT_XMIT_Q_HEADER
WebSphere® Support Technical Exchange
15
IBM Software Group
Customer Scenario 1
iSeries/Sender
WMQ Level 5.3
VSE/Receiver
WMQ Level 2.1.2
OS Level 530
OS Level ???
QMGR CCSID 37
QMGR CCSID 1047
Sender Convert *YES
Problem : Sender channel fails with message AMQ9541
Message . . . . :
CCSID supplied for data conversion not supported.
Cause . . . . . :
The program ended because, either the source CCSID '&1'
or the target CCSID '&2' is not valid, or is not currently supported.
Recovery . . . :
Correct the CCSID that is not valid, or ensure that the
requested CCSID can be supported.
WebSphere® Support Technical Exchange
16
IBM Software Group
Customer Scenario 1 cont…
Solution : At the time this issue was submitted the
customer had the following options
Change CONVERT to *NO on the Sender channel
Change VSE QMGR CCSID to 924/37
As of May/2007 CCSID 1047 has now been added,
along with CCSIDs 5346, 5347, 5349, 5350, 5351,
5352, and 5353 to the iSeries server. Reference apar
SE28865 (PTF SI277906 for OS/530 and SI27516 for
OS/540).
WebSphere® Support Technical Exchange
17
IBM Software Group
Customer Scenario 2
iSeries/Sender
WMQ Level 5.3.12
zOS/Receiver
WMQ Level 5.3.1
OS Level 530
OS Level ???
QMGR CCSID 37
QMGR CCSID 5026
Sender Convert *NO
Problem : Enduser indicated that they have an iSeries
communicating with z/OS. The sender channel is going into
RETRYING status and message AMQ9503 (Channel
negotiation failed) is generated when the PNGMQMCHL is
issued.
WebSphere® Support Technical Exchange
18
IBM Software Group
Customer Scenario 2 cont…
Solution : Found that the receiver/responder
determined it could not handle the Fap level and/or
encoding scheme being negotiated between the
channels.
Work-Around
Implement an intermediate queue manager
• iSeries (CCSID 37) z/OS (CCSID 5035)
z/OS (CCSID 5026)
WebSphere® Support Technical Exchange
19
IBM Software Group
Customer Scenario 2 cont…
Work-Around
Implement server/requester channel pair
• iSeries (CCSID 37) z/OS (CCSID 5026)

• This configuration will only work as long as the
initiator is the z/OS
• WMQ z/OS L3 team did find that this was a
defect and created a usermod, however it was
not verified by the customer as they decided to
continue to function with the intermediate queue
manger in place.
WebSphere® Support Technical Exchange
20
IBM Software Group
What Gets Converted,?
For Channel Communication -- When a channel between two
WebSphere MQ Queue Managers is started, the two Queue Managers
will negotiate which CCSID and encoding protocol they will use for
channel communications.
Reference IBM Technote 7005729 : Data Conversion under
WebSphere MQ
WebSphere® Support Technical Exchange
21
IBM Software Group
Customer Scenario 3
Windows/Sender
WMQ Level 5.3.10
iSeries/Receiver
WMQ Level 5.3.11
OS Level ???
OS Level ???
QMGR CCSID 437
QMGR CCSID 500
Problem : Characters |, ] and ! are not being properly
handled when sent from Windows to iSeries.
WebSphere® Support Technical Exchange
22
IBM Software Group
Customer Scenario 3 cont…
Solution : Change the queue manager on iSeries to
CCSID 37 or a create a channel exit program that will
then handle the conversion.
Reference IBM Technote 1207860 : Conversion
character difference between CCSID 37 and 500
WebSphere® Support Technical Exchange
23
IBM Software Group
Code page 37 - Code page 500 differences
WebSphere® Support Technical Exchange
24
IBM Software Group
Common ‘problem’ code points
Character
CCSID 437
CCSID 37
CCSID 500
!
A5
5A
4F
¬
F5
5F
BA
¢
A4
4A
B0
|
F4
4F
BB
[
AB
BA
A4
]
BB
BB
A5
^
0B
B0
5F
WebSphere® Support Technical Exchange
25
IBM Software Group
Customer Scenario 4
Sun Solaris/WMQ Client
WMQ Level 5.3.12
z/OS /SVRCONN and RCVR
WMQ Level 5.3.12
OS Level ???
OS Level ???
Putting Application CCSID 819
QMGR CCSID 500
z/OS / Sender
WMQ Level
OS Level ???
QMGR CCSID 500
Windows / WMQ Client
WMQ Level
OS Level
Getting Application CCSID 1252
WebSphere® Support Technical Exchange
Must have Client attachment
Problem: Messages put to z/OS
queue with a CCSID of 819 are
processed properly by the
Getting client (Windows)
application. However,
messages put by the z/OS
queue manager with CCSID of
500 are not converted by getting
client application.
26
IBM Software Group
Customer Scenario 4 cont…
Solution : Messages PUT from the z/OS queue
manager or any platform should have MQFMT_STRING
format specified in the message header. Also the
Getting Client application should make sure it has
CCSID 437 specified, along with MQGMO option
Convert.
Reference: WebSphere MQ Application Programming
Reference (SC34-6062)
WebSphere® Support Technical Exchange
27
IBM Software Group
The message is entirely character data
If the message you are passing is entirely character data then it is
very easy to have WebSphere MQ do all of the work for you. In the
application that is doing the MQPUT of the message, you should
set MQMD.Format to MQFMT_STRING. MQFMT_STRING is a
constant that equates to the string value “MQSTR “. This value in
the Format field will let WebSphere MQ know that the message
consists entirely of character string data.
Reference IBM Technote 7005729 : Data Conversion
under WebSphere MQ
WebSphere® Support Technical Exchange
28
IBM Software Group
Common Procedure to ensure conversion on a
MQGET
In the application doing the MQGET of the message, set MQMD.CodedCharSetId to
0 to pick up the default CCSID of the QMGR. This will ensure that WebSphere MQ
knows the correct CCSID to convert to.
In the application that is doing the MQGET of the message, make sure you specify
MQGMO_CONVERT as one of the MQGMO.Options. This field has many options
that should be either added or bitwise OR’ed together if you need to specify more
than one.
It is recommended that you always do the data conversion on the MQGET of the
message, since the message may take many ‘hops’ across different platforms on its
way to its destination, and doing data conversion on each ‘hop’ is a waste of
resources.
If you specify CONVERT(YES) on the SENDER channel that sends the data to its
final destination, WebSphere MQ will convert the data into the CCSID of the queue
manager on the target system.
WebSphere® Support Technical Exchange
29
IBM Software Group
Customer Scenario 5
iSeries /SDR
WMQ Level 5.3.12
OS Level 5.2
QMGR CCSID 420
Tandem /RCVR
WMQ Level 5.1
OS Level K1000
QMGR CCSID 1089
Windows / RCVR
WMQ Level 6.0.0.0
OS Level 2003 SP1
QMGR CCSID 1256
Problem: When a message is put with Arabic character LamAlif it is not being converted properly between CCSID 420 and
CCISD 1089(Tandem) or 1256(Windows).
WebSphere® Support Technical Exchange
30
IBM Software Group
Customer Scenario 5 cont…
Solution :This is not a character conversion limitation for
WebSphere MQ, as not all characters can be converted
from one CCSID to another. In this case the Lam-Alif
character does not exist in CCSID/Code Page 1089 nor
1256, so a substitution character would be used, for
example X’1A.
Enduser would need to specify a user defined message
format and write their own conversion code.
Reference: WebSphere MQ Application Programming
Reference (SC34-6062)
WebSphere® Support Technical Exchange
31
IBM Software Group
Problem Determination
Complete problem description
WMQ Environment description
WMQ Level, for example WMQ 6.0.2.2
OS level, for example iSeries 5.3.0; Windows XP SP2
QMGR CCSID, for example 37, 437, 5026, 1047
QMGR Sender Channel CONVERT setting
Putting Application MQMD CCSID and Format setting
Getting Application MQMD CCSID setting
Capture the message on the XMITQ.
• Verify if it is correct.
Capture the message on the Destination Queue.
• Verify if it is correct
WMQ Trace on both sending and receiving servers
IBM MustGather Technote : 1176955
WebSphere® Support Technical Exchange
32
IBM Software Group
IBM Technotes
MustGather: Documentation required by
WebSphere MQ – data conversion
1176955
Data Conversion under WebSphere MQ
7005729
Conversion character differences between
CCSID 037 and CCSID 500
1207860
Data conversion in MQ intermittently does
not behave as expected
1240018
EBCDIC NL character is not converting
correctly in ASCII 0x15 (EBCDIC NL) converts to 0xDA
1116311
AMQ6174 MQRC 2119 putting a message
to a remote queue
1109867
WebSphere® Support Technical Exchange
33
IBM Software Group
IBM Technotes cont…
2110 MQRC Format Error
1232346
2080 MQRC TRUNCATED MSG FAILED
1167205
Data conversion does not occur in a
clustering environment
1268752
What CCSID is used by default for
WebSphere MQ client messages
1215215
MQ data conversion and data conversion
exit programs
1167007
WebSphere® Support Technical Exchange
34
IBM Software Group
Useful Links
WebSphere MQ Support
www.ibm.com/software/integration/wmq/support
System i Globalization
http://www.ibm.com/servers/eserver/iseries/software/globali
zation/
Unicode Home Page
http://www.unicode.org/
Character Sets and Code Pages, Encoding
http://www.i18nguy.com/unicode/codepages.html
Language Tools
http://www.google.com/language_tools?hl=en
http://babelfish.altavista.com/
WebSphere® Support Technical Exchange
35
IBM Software Group
Useful Tools
WebSphere MQ Supportpac
IH03 (IH03: WebSphere Message Broker V6-Message display,
test & performance utilities)
• IBM Technote 4000637
• RFHUTIL and RFHUTILC allows enduser to place test message
to a queue with a particular CCSID
WebSphere MQ Sample
AMQSBCG (distributed (Windows and Unix))
AMQSBCG4 (iSeries)
• These programs are passed the name of a queue and a queue
manager. It then reads each message from the queue and
outputs the following to the stdout
− Formatted message descriptor fields
− Message data (dumped in hex and, where possible,
character format)
WebSphere® Support Technical Exchange
36
IBM Software Group
AMQSBCG/AMQSBCG4
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : '
'
Priority : 0 Persistence : 0
MsgId : X'C1D4D840C3D6D4D4D6D540404040404046D2D73320001402'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ
:'
'
ReplyToQMgr : 'COMMON
'
** Identity Context
UserIdentifier : 'STANLEY '
AccountingToken :
X'0F40404040404040404040404040404000000000000000000000000000
000008'
ApplIdentityData : '
'
WebSphere® Support Technical Exchange
37
IBM Software Group
AMQSBCG/AMQSBCG4 cont…
** Origin Context
PutApplType : '8'
PutApplName : '021195/STANLEY/QP0ZSPWT
PutDate : '20070903' PutTime : '18330499'
ApplOriginData : ' '
'
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset
: '0'
MsgFlags
: '0'
OriginalLength : '-1'
**** Message
00000000:
00000010:
00000020:
00000030:
**** length - 52 bytes
4A61 7661 2076 6572 7369 6F6E 206F 6620 '¢/Î/.ÎÁÊËÑ?>.?Ã.'
7468 6520 4D51 5055 5431 2061 7069 2020 'ÈÇÁ.(é&íè../øÑ..'
203A 20A3 A331 302E 3030 2024 2420 A2A2 '...tt.........ss'
2023 2320
'....
'
WebSphere® Support Technical Exchange
38
IBM Software Group
z/OS Browse Sample
WebSphere MQ Support pac
MA17 (WebSphere MQ for z/OS – Message
Handler Sample)
• IBM Technote 4000071
WebSphere® Support Technical Exchange
39
IBM Software Group
z/OS MA17 browse sample
Queue Manager : RTP4
Queue
: RTP4.QL
Forward to Q Mgr : RTP6
Forward to Queue :
:
:
:
:
Action : : (D)elete (F)orward
Message Content :
-----------------------------------------------------------------------------Message Descriptor
StrucId
: `MD `
Version
: 000000001
Report
: 000000000
MsgType
: 000000008
Expiry
: -00000001
Feedback
: 000000000
Encoding
: 000000785
CodedCharSetId : 000000037
Format
: `MQSTR `
Priority
: 000000000
Persistence
: 000000000
MsgId
: `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
CorrelId
: `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
BackoutCount : 000000002
ReplyToQ
:`
`
ReplyToQMgr
: `RTP6
`
WebSphere® Support Technical Exchange
40
IBM Software Group
Documentation
The following manuals discuss MQ conversion topics. Just refer to
the index of these manuals for ‘conversion’ or ‘CCSID’
WMQ Commands Manual Reference
WMQ Application Programming Reference
WMQ Application Programming Guide
Sample programs documented in this manual
WMQ Intercommunication
WMQ Conversion document
All manuals and documents can be found on the WebSphere MQ
Support webpage:
www.ibm.com/software/integration/wmq/support
WebSphere® Support Technical Exchange
41
IBM Software Group
CCSIDs specified in this presentation
CCSID
Description
37
USA, Canada, Netherlands, Portugal, Brazil, Australia, New
Zealand
420
Arabic EBCDIC
437
USA PC-Data
500
International EBCDIC
819
ISO 8859-1 ASCII
1047
Latin 1 (Open Systems)
1089
Arabic ISO 8859-6
1252
MS-WIN Latin-1
1256
MS-WIN Arabic
WebSphere® Support Technical Exchange
42
IBM Software Group
Summary
Terms and Definitions
Conversion vs. Translation
CCSID (Coded Character Set Identifier)
Encoding
IBM Technotes
Useful Links
WebSphere MQ Product Support page
• www.ibm.com/software/integration/wmq/support
• Search on Data Conversion to review IBM
Technotes regarding this topic
WebSphere® Support Technical Exchange
43
IBM Software Group
Additional WebSphere Product Resources
Discover the latest trends in WebSphere Technology and implementation,
participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/
Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html
Join the Global WebSphere User Group Community: www.websphere.org
Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant
Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html
Sign up to receive weekly technical My support emails:
www.ibm.com/software/support/einfo.html
WebSphere® Support Technical Exchange
44
IBM Software Group
Questions and Answers
WebSphere® Support Technical Exchange
45