1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>codecvt</title><meta name="generator" content="DocBook XSL Stylesheets V1.74.0" /><meta name="keywords" content=" ISO C++ , codecvt " /><meta name="keywords" content=" ISO C++ , library " /><link rel="home" href="../spine.html" title="The GNU C++ Library Documentation" /><link rel="up" href="facets.html" title="Chapter 15. Facets aka Categories" /><link rel="prev" href="facets.html" title="Chapter 15. Facets aka Categories" /><link rel="next" href="messages.html" title="messages" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">codecvt</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="facets.html">Prev</a> </td><th width="60%" align="center">Chapter 15. Facets aka Categories</th><td width="20%" align="right"> <a accesskey="n" href="messages.html">Next</a></td></tr></table><hr /></div><div class="sect1" lang="en" xml:lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="manual.localization.facet.codecvt"></a>codecvt</h2></div></div></div><p>
4 The standard class codecvt attempts to address conversions between
5 different character encoding schemes. In particular, the standard
6 attempts to detail conversions between the implementation-defined wide
7 characters (hereafter referred to as wchar_t) and the standard type
8 char that is so beloved in classic “<span class="quote">C</span>” (which can now be
9 referred to as narrow characters.) This document attempts to describe
10 how the GNU libstdc++ implementation deals with the conversion between
11 wide and narrow characters, and also presents a framework for dealing
12 with the huge number of other encodings that iconv can convert,
13 including Unicode and UTF8. Design issues and requirements are
14 addressed, and examples of correct usage for both the required
15 specializations for wide and narrow characters and the
16 implementation-provided extended functionality are given.
17 </p><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.req"></a>Requirements</h3></div></div></div><p>
18 Around page 425 of the C++ Standard, this charming heading comes into view:
19 </p><div class="blockquote"><blockquote class="blockquote"><p>
20 22.2.1.5 - Template class codecvt
21 </p></blockquote></div><p>
22 The text around the codecvt definition gives some clues:
23 </p><div class="blockquote"><blockquote class="blockquote"><p>
24 <span class="emphasis"><em>
25 -1- The class codecvt<internT,externT,stateT> is for use when
26 converting from one codeset to another, such as from wide characters
27 to multibyte characters, between wide character encodings such as
30 </p></blockquote></div><p>
31 Hmm. So, in some unspecified way, Unicode encodings and
32 translations between other character sets should be handled by this
34 </p><div class="blockquote"><blockquote class="blockquote"><p>
35 <span class="emphasis"><em>
36 -2- The stateT argument selects the pair of codesets being mapped between.
38 </p></blockquote></div><p>
39 Ah ha! Another clue...
40 </p><div class="blockquote"><blockquote class="blockquote"><p>
41 <span class="emphasis"><em>
42 -3- The instantiations required in the Table ??
43 (lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and
44 codecvt<char,char,mbstate_t>, convert the implementation-defined
45 native character set. codecvt<char,char,mbstate_t> implements a
46 degenerate conversion; it does not convert at
47 all. codecvt<wchar_t,char,mbstate_t> converts between the native
48 character sets for tiny and wide characters. Instantiations on
49 mbstate_t perform conversion between encodings known to the library
50 implementor. Other encodings can be converted by specializing on a
51 user-defined stateT type. The stateT object can contain any state that
52 is useful to communicate to or from the specialized do_convert member.
54 </p></blockquote></div><p>
55 At this point, a couple points become clear:
57 One: The standard clearly implies that attempts to add non-required
58 (yet useful and widely used) conversions need to do so through the
59 third template parameter, stateT.</p><p>
60 Two: The required conversions, by specifying mbstate_t as the third
61 template parameter, imply an implementation strategy that is mostly
62 (or wholly) based on the underlying C library, and the functions
63 mcsrtombs and wcsrtombs in particular.</p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.design"></a>Design</h3></div></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="codecvt.design.wchar_t_size"></a><span class="type">wchar_t</span> Size</h4></div></div></div><p>
64 The simple implementation detail of wchar_t's size seems to
65 repeatedly confound people. Many systems use a two byte,
66 unsigned integral type to represent wide characters, and use an
67 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
68 Java, others.) Other systems, use a four byte, unsigned integral
69 type to represent wide characters, and use an internal encoding
70 of UCS4. (GNU/Linux systems using glibc, in particular.) The C
71 programming language (and thus C++) does not specify a specific
72 size for the type wchar_t.
74 Thus, portable C++ code cannot assume a byte size (or endianness) either.
75 </p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="codecvt.design.unicode"></a>Support for Unicode</h4></div></div></div><p>
76 Probably the most frequently asked question about code conversion
77 is: "So dudes, what's the deal with Unicode strings?"
78 The dude part is optional, but apparently the usefulness of
79 Unicode strings is pretty widely appreciated. Sadly, this specific
80 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
81 etc etc etc) are not mentioned in the C++ standard.
85 The thought that all one needs to convert between two arbitrary
86 codesets is two types and some kind of state argument is
87 unfortunate. In particular, encodings may be stateless. The naming
88 of the third parameter as stateT is unfortunate, as what is really
89 needed is some kind of generalized type that accounts for the
90 issues that abstract encodings will need. The minimum information
91 that is required includes:
92 </p><div class="itemizedlist"><ul type="disc"><li><p>
93 Identifiers for each of the codesets involved in the
94 conversion. For example, using the iconv family of functions
95 from the Single Unix Specification (what used to be called
96 X/Open) hosted on the GNU/Linux operating system allows
97 bi-directional mapping between far more than the following
98 tantalizing possibilities:
100 (An edited list taken from <code class="code">`iconv --list`</code> on a
101 Red Hat 6.2/Intel system:
102 </p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting">
103 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
104 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
105 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
106 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
107 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
108 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
109 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
110 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
111 UTF-16, UTF8, UTF16).
112 </pre></blockquote></div><p>
113 For iconv-based implementations, string literals for each of the
114 encodings (i.e. "UCS-2" and "UTF-8") are necessary,
116 non-iconv implementations a table of enumerated values or some other
117 mechanism may be required.
119 Maximum length of the identifying string literal.
121 Some encodings require explicit endian-ness. As such, some kind
122 of endian marker or other byte-order marker will be necessary. See
123 "Footnotes for C/C++ developers" in Haible for more information on
124 UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
125 however implementations, most notably Microsoft, vary.)
127 Types representing the conversion state, for conversions involving
128 the machinery in the "C" library, or the conversion descriptor, for
129 conversions using iconv (such as the type iconv_t.) Note that the
130 conversion descriptor encodes more information than a simple encoding
133 Conversion descriptors for both directions of encoding. (i.e., both
134 UCS-2 to UTF-8 and UTF-8 to UCS-2.)
136 Something to indicate if the conversion requested if valid.
138 Something to represent if the conversion descriptors are valid.
140 Some way to enforce strict type checking on the internal and
141 external types. As part of this, the size of the internal and
142 external types will need to be known.
143 </p></li></ul></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="codecvt.design.issues"></a>Other Issues</h4></div></div></div><p>
144 In addition, multi-threaded and multi-locale environments also impact
145 the design and requirements for code conversions. In particular, they
146 affect the required specialization codecvt<wchar_t, char, mbstate_t>
147 when implemented using standard "C" functions.
149 Three problems arise, one big, one of medium importance, and one small.
151 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
152 on all systems required by the GNU tools. For GNU/Linux and glibc,
153 this is not an issue.
155 Of medium concern, in the grand scope of things, is that the functions
156 used to implement this specialization work on null-terminated
157 strings. Buffers, especially file buffers, may not be null-terminated,
158 thus giving conversions that end prematurely or are otherwise
161 The last, and fundamental problem, is the assumption of a global
162 locale for all the "C" functions referenced above. For something like
163 C++ iostreams (where codecvt is explicitly used) the notion of
164 multiple locales is fundamental. In practice, most users may not run
165 into this limitation. However, as a quality of implementation issue,
166 the GNU C++ library would like to offer a solution that allows
167 multiple locales and or simultaneous usage with computationally
168 correct results. In short, libstdc++ is trying to offer, as an
169 option, a high-quality implementation, damn the additional complexity!
171 For the required specialization codecvt<wchar_t, char, mbstate_t> ,
172 conversions are made between the internal character set (always UCS4
173 on GNU/Linux) and whatever the currently selected locale for the
174 LC_CTYPE category implements.
175 </p></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.impl"></a>Implementation</h3></div></div></div><p>
176 The two required specializations are implemented as follows:
179 codecvt<char, char, mbstate_t>
182 This is a degenerate (i.e., does nothing) specialization. Implementing
183 this was a piece of cake.
186 codecvt<char, wchar_t, mbstate_t>
189 This specialization, by specifying all the template parameters, pretty
190 much ties the hands of implementors. As such, the implementation is
191 straightforward, involving mcsrtombs for the conversions between char
192 to wchar_t and wcsrtombs for conversions between wchar_t and char.
194 Neither of these two required specializations deals with Unicode
195 characters. As such, libstdc++ implements a partial specialization
196 of the codecvt class with and iconv wrapper class, encoding_state as the
197 third template parameter.
199 This implementation should be standards conformant. First of all, the
200 standard explicitly points out that instantiations on the third
201 template parameter, stateT, are the proper way to implement
202 non-required conversions. Second of all, the standard says (in Chapter
203 17) that partial specializations of required classes are a-ok. Third
204 of all, the requirements for the stateT type elsewhere in the standard
205 (see 21.1.2 traits typedefs) only indicate that this type be copy
208 As such, the type encoding_state is defined as a non-templatized, POD
209 type to be used as the third type of a codecvt instantiation. This
210 type is just a wrapper class for iconv, and provides an easy interface
211 to iconv functionality.
213 There are two constructors for encoding_state:
216 encoding_state() : __in_desc(0), __out_desc(0)
219 This default constructor sets the internal encoding to some default
220 (currently UCS4) and the external encoding to whatever is returned by
221 nl_langinfo(CODESET).
224 encoding_state(const char* __int, const char* __ext)
227 This constructor takes as parameters string literals that indicate the
228 desired internal and external encoding. There are no defaults for
231 One of the issues with iconv is that the string literals identifying
232 conversions are not standardized. Because of this, the thought of
233 mandating and or enforcing some set of pre-determined valid
234 identifiers seems iffy: thus, a more practical (and non-migraine
235 inducing) strategy was implemented: end-users can specify any string
236 (subject to a pre-determined length qualifier, currently 32 bytes) for
237 encodings. It is up to the user to make sure that these strings are
238 valid on the target system.
245 Strangely enough, this member function attempts to open conversion
246 descriptors for a given encoding_state object. If the conversion
247 descriptors are not valid, the conversion descriptors returned will
248 not be valid and the resulting calls to the codecvt conversion
249 functions will return error.
256 Provides a way to see if the given encoding_state object has been
257 properly initialized. If the string literals describing the desired
258 internal and external encoding are not valid, initialization will
259 fail, and this will return false. If the internal and external
260 encodings are valid, but iconv_open could not allocate conversion
261 descriptors, this will also return false. Otherwise, the object is
262 ready to convert and will return true.
265 encoding_state(const encoding_state&)
268 As iconv allocates memory and sets up conversion descriptors, the copy
269 constructor can only copy the member data pertaining to the internal
270 and external code conversions, and not the conversion descriptors
273 Definitions for all the required codecvt member functions are provided
274 for this specialization, and usage of codecvt<internal character type,
275 external character type, encoding_state> is consistent with other
277 </p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.use"></a>Use</h3></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting">
278 typedef codecvt_base::result result;
279 typedef unsigned short unicode_t;
280 typedef unicode_t int_type;
281 typedef char ext_type;
282 typedef encoding_state state_type;
283 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt;
285 const ext_type* e_lit = "black pearl jasmine tea";
286 int size = strlen(e_lit);
287 int_type i_lit_base[24] =
288 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
289 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
292 const int_type* i_lit = i_lit_base;
293 const ext_type* efrom_next;
294 const int_type* ifrom_next;
295 ext_type* e_arr = new ext_type[size + 1];
297 int_type* i_arr = new int_type[size + 1];
300 // construct a locale object with the specialized facet.
301 locale loc(locale::classic(), new unicode_codecvt);
302 // sanity check the constructed locale has the specialized facet.
303 VERIFY( has_facet<unicode_codecvt>(loc) );
304 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
305 // convert between const char* and unicode strings
306 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
307 initialize_state(state01);
308 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
309 i_arr, i_arr + size, ito_next);
310 VERIFY( r1 == codecvt_base::ok );
311 VERIFY( !int_traits::compare(i_arr, i_lit, size) );
312 VERIFY( efrom_next == e_lit + size );
313 VERIFY( ito_next == i_arr + size );
314 </pre></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.future"></a>Future</h3></div></div></div><div class="itemizedlist"><ul type="disc"><li><p>
315 a. things that are sketchy, or remain unimplemented:
316 do_encoding, max_length and length member functions
317 are only weakly implemented. I have no idea how to do
318 this correctly, and in a generic manner. Nathan?
320 b. conversions involving std::string
321 </p><div class="itemizedlist"><ul type="circle"><li><p>
322 how should operators != and == work for string of
323 different/same encoding?
325 what is equal? A byte by byte comparison or an
326 encoding then byte comparison?
328 conversions between narrow, wide, and unicode strings
329 </p></li></ul></div></li><li><p>
330 c. conversions involving std::filebuf and std::ostream
331 </p><div class="itemizedlist"><ul type="circle"><li><p>
332 how to initialize the state object in a
333 standards-conformant manner?
335 how to synchronize the "C" and "C++"
336 conversion information?
338 wchar_t/char internal buffers and conversions between
339 internal/external buffers?
340 </p></li></ul></div></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h3 class="title"><a id="facet.codecvt.biblio"></a>Bibliography</h3></div></div></div><div class="biblioentry"><a id="id415012"></a><p><span class="title"><i>
342 </i>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a id="id514935"></a><p><span class="title"><i>
344 </i>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id416284"></a><p><span class="title"><i>
345 ISO/IEC 14882:1998 Programming languages - C++
346 </i>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id416302"></a><p><span class="title"><i>
347 ISO/IEC 9899:1999 Programming languages - C
348 </i>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="id459727"></a><p><span class="title"><i>
349 System Interface Definitions, Issue 6 (IEEE Std. 1003.1-200x)
350 </i>. </span><span class="copyright">Copyright © 1999
351 The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span><span class="biblioid">
352 <a class="ulink" href="http://www.opennc.org/austin/docreg.html" target="_top">
354 . </span></p></div><div class="biblioentry"><a id="id459755"></a><p><span class="title"><i>
355 The C++ Programming Language, Special Edition
356 </i>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
358 . </span></span></p></div><div class="biblioentry"><a id="id430786"></a><p><span class="title"><i>
359 Standard C++ IOStreams and Locales
360 </i>. </span><span class="subtitle">
361 Advanced Programmer's Guide and Reference
362 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
363 Addison Wesley Longman
364 . </span></span></p></div><div class="biblioentry"><a id="id407191"></a><p><span class="title"><i>
365 A brief description of Normative Addendum 1
366 </i>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span><span class="biblioid">
367 <a class="ulink" href="http://www.lysator.liu.se/c/na1.html" target="_top">
369 . </span></p></div><div class="biblioentry"><a id="id394112"></a><p><span class="title"><i>
371 </i>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span><span class="biblioid">
372 <a class="ulink" href="ftp://ftp.ilog.fr/pub/Users/haible/utf8/Unicode-HOWTO.html" target="_top">
374 . </span></p></div><div class="biblioentry"><a id="id394140"></a><p><span class="title"><i>
375 UTF-8 and Unicode FAQ for Unix/Linux
376 </i>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span><span class="biblioid">
377 <a class="ulink" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html" target="_top">
379 . </span></p></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="facets.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="facets.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="messages.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 15. Facets aka Categories </td><td width="20%" align="center"><a accesskey="h" href="../spine.html">Home</a></td><td width="40%" align="right" valign="top"> messages</td></tr></table></div></body></html>
projects / l4.git / blob