Project team report

CEN/TC304/PT01

User requirements study and programming for standardization in the field of Character Set Technology

or: Standards for providing multi-cultural functionality in IT-systems

Final report
1995-09-06

for approval by:
CEN, CENELEC, ETSI and EWOS

Editor: Ţorvarđur Kári Ólafsson

FOREWORD

This report was developed by a Project team that was set up under mandate M/037 from EC to develop a European strategy for Character Set Technology (CST) standardization. The work is a direct continuation of the CST workshop that was held in Luxemburg 1-2 December 1994 [1].

The final report is submitted to CENELEC, ETSI, EWOS and CEN for approval. CEN/TC304 is responsible for its final approval.

CONTENTS

FOREWORD

PART 0 -- GENERAL

PART I -- USER REQUIREMENTS STUDY

PART II -- TAXONOMY

PART III -- STRATEGY

PART IV -- WORK NEEDED FOR MULTI-CULTURAL SUPPORT IN IT

ANNEXES

A. Members of this PT

B. Bibliography

C. Mandate M/037

D. References to standards

E. General recommendations of the CST-workshop

F. The indigenous languages of Europe

G. Extract from the Nordic report on cultural requirements

PART 0 -- General

1. Introduction and background

The work of other organizations then started to affect the future of CST standardization in Europe:

• In 1991 and 1992, EWOS published two Technical Reports on the usage of coded character sets and repertoires in EWOS.
  
• In 1992, STRÍ published a report on Nordic cultural requirements on Information Technology. This report gave the first structured overview of the internationalization problems related to CST.
  
• In 1993, an international standard on multi-byte-coded character sets was published as ISO/IEC 10646-1. The so-called Unicode, already implemented by some of the major manufacturers, is an option of this standard.
  
• In 1992-3, ITSTC disbanded its own standards development groups and moved the work to CEN, CENELEC and ETSI.
  

This development led to the disbanding of IT/WG-CSC and the establishment of CEN/TC304 in late 1992. The new committee has an extended scope, derived from a first draft taxonomy that was based on the Nordic report on cultural requirements.

In December 1993, the EC Senior Official Group on IT standardization (SOGITS) through its secretariat in EC DG.III, issued a standardization mandate in the field of CST, asking CEN/TC304 to develop a European strategy for CST standardization.

In order to execute this mandate, TC304 organized a workshop and established the Project Team PT01. The workshop was on European User Requirements for Internationalization of IT and Character Set Technology and took place in Luxemburg 1-2 December 1994. The Project Team is basing its work on the results of this workshop [1], see further Part I and Annex E.

The ultimate goal of this work is that new CST standards meet user requirements and provide basis for a true multilingual infrastructure in Europe. A further goal is that users know which standards are available and may obtain guidance on their usage.

2. Scope

The report defines a European policy and strategy with respect to CST, aligns the existing and ongoing standardization activities with this policy, and finally it defines a European work program for CST, including a taxonomy and European sub-repertoires of ISO/IEC 10646. This work is based on a clear definition of European user requirements. All required standardization activities in order to realise the user functionality should be defined.

See also Annex C.

3. Definitions, abbreviations, symbols

4. Conclusions and recommendations

4.1 Conclusions

The project team has met the requirements of the mandate (see Annex C). A joint European work programme for standardization in the field of CST has been agreed between CEN, CENELEC, ETSI and EWOS (see Part IV). This work programme has been based on a clear definition of user requirements (see Part I). These user requirements were gathered at the open workshop held in Luxemburg in December 1994. A taxonomy for CST is presented in Part II. A strategy for implementation of it is identified in Part III which involves not just standardization but also R-and-D, promotion and coordination activities.

4.2 Recommendations

Recommendation 1

Recommendation 2

Recommendation 3

PART I -- User requirements study

1. Introduction and scope

This part aims to define which are the requirements for CST standardization in Europe on part of the prospective users of the standards. The users include other standardizers, End users, IT applications, IT industry, service providers and public procurers.

For the purposes of this study, user requirements are grouped in an order of priority, ranging from High through Medium to Low. A high priority means that the topic should be addressed urgently in order to improve the usability of IT for most people. Medium priority is assigned when the action would improve the usability of IT for some people, or improve it marginally for most users. Low priority topics would not improve usability for most users, but general benefits would accrue, or some selected groups of users would benefit marginally.

Note that this part of the report tries to describe the user requirements regardless of whether they are to do with standardization or not. The standardization perspective is introduced in Part III.

2. Definitions

Attributes: those general aspects of a character used independently of design (font or style), to convey additional abstract meaning, e.g. bold or underline.

Character: a member of a set of elements used for the organization, control or representation of data [ISO 4873]. NOTE: Characters are sub-divided into graphic characters and control characters.

Character set (Character repertoire): a finite set of different characters that is complete for a given purpose [ISO 2382-4].

Character Set Technology: the technology for handling characters within IT and providing multi-cultural functionality in informatic and telematic systems. This includes input, coding, interchange, rendition, transformation, identification, ordering and other ways of manipulating character data by electronic means.

Charmap: a text file describing a coded character set. Each character set description file defines characteristics for a coded character set and the encoding of characters. Other information about the coded character set may also be in the file. Coded character set character values are defined using symbolic character names followed by character encoding values [ISO/IEC 9945-2].

Coded character set, Code: a set of unambiguous rules that establishes a character set and the one-to-one relationship between the characters of the set and their bit combinations [ISO 4873].

Coding scheme: a collection of rules that maps the elements of one set onto the elements of a second set.

Coding: the process of allocating unique bit combinations to an individual character according to established rules.

Combining character: a member of an identified subset of the coded character set of ISO/IEC 10646 intended for combination with the preceding non-combining graphic character, or with a sequence of combining characters preceded by a non-combining character [ISO/IEC 10646].

Control function: an action that affects the recording, processing, transmission or interpretation of data, and that has a coded representation consisting of one or more bit combinations [ISO 6429].

Cultural convention: a convention of an information system which is functionally common between regional cultures, but may differ in presentation, operation behaviour or degree of importance. NOTE: application or organizational cultures are not considered here [3].

Cultural register: a register of cultural conventions related to IT.

Cultural requirements: requirements due to properties of the language(s), commonly accepted rules for its use -- especially in written form -- or other special characteristics of a society in a certain geographic area [5].

Diacritical mark: a combining character which forms part of a letter.

Europe: the geographic area whose boundaries are the Ural Mountains, the Caspian, Black and Mediterranian Seas, the Atlantic Ridge and the North Pole.

Fallback representation: an approximate representation of a character, made for equipment not capable of representing the character correctly.

Font: a collection of glyph images having the same basic design, e.g. Courier Bold Oblique [ISO 9541-1].

Glyph: a recognisable abstract graphic symbol which is independent of any specific design [ISO 9541-1].

Graphic character: a character, other than a control function, that has a visual representation, normally hand-written, printed or displayed, and has a coded representation consisting of one or more bit combinations [ISO 4873].

Indigenous: native to a certain geographic area.

Internationalization: a process of producing an application platform or application which is capable of being localized for (almost) any cultural environmental easily. NOTE: an internationalized information system does not have a dependency on any specific culture unless it is localized to that selected culture [3].

Letter: a graphic character that is in the alphabet of a natural language [5].

Locale: the definition of the environment of a user that depends on language and cultural conventions. It is made up from one or more categories. Each category is identified by its name and controls specific aspects of the behaviour of components of the system [ISO/IEC 9945-2].

Localization: a process of adapting an internationalized application to a specific cultural environment [5].

Operating system: software that controls the execution of programs and that may provide services such as resource allocation, scheduling, input/output control and data management [ISO 2382-1].

Ordering: an operation by which two different objects (for example two character strings) are assigned a context-free deterministic ordering [3].

Script: a set of graphic characters used for the written form of one or more languages [3]. Examples: Latin script, Cyrillic script, Greek script.

Special character: a graphic character that is not a letter, a digit or a spacing character [ISO 2382-4].

Symbol: a character or letter, or stylized representation of an object.

Taxonomy: a classification of concepts or a terminology system.

Telematics: the application of information and communications technologies and services, usually in direct combination.

Transcription: the process whereby the pronunciation of a given language is noted by the system of signs of a conversion language. A transcription system is of necessity based on the orthographic conventions of the conversion language. Transcription is not strictly reversible [ISO 3602].

Transformation: any conversion of coded character data, including transliteration, transcription, code conversion and fallback rules.

Transliteration: the process which consists of representing the characters of an alphabetical or syllable writing by the characters of a conversion alphabet. In principle, this conversion should be made character by character [ISO 3602]. NOTE: transliteration is a reversible process.

User interface: the part of a system with which the user interacts [ANSI X3.172].

Users (1. of IT) Persons or organizations utilising IT in their work or leisure (2. of CST-standards) Users of IT , providers of IT-technology, developers of IT-standards.

3. Abbreviations

4. Statement of problem -- the Users' requirements

4.1 The users

It is convenient to consider a number of large easily identifiable groups of users and then to examine their needs in relation to applications and tasks they need to perform.

4.1.1 The working population

A rapidly increasing percentage of these working people will make use of IT in some form or another, whether directly with word processing, or indirectly using data bases or data processing. Their tasks will all involve reading input and output via characters. To a large extent these characters will be the ones in their local language, but at some time or other many of the users will have the need to interact with other users whose language or culture uses an alternative character set.

To avoid incomprehension, or at least inefficiency, they must be able to access these "foreign", unfamiliar characters. Additionally, users will interact with IT "tools", e.g. software which may present feedback and help information in another language. Neither the tools nor the medium must impose any barrier to users' efficiency. Users' requirements are for access to own language scripts and fonts and translation from/to others for interaction with other countries, nationalities and cultures.

Users in the working population are characterized by aptitudes, training and experience which generally fit the task they perform. That task may demand a high degree of "task fit", in cultural and language terms. Different and highly specific user requirements may often be identified within user groups or professions, such as authors and writers (technical and scientific, fiction), journalists, translators and interpreters, engineers, scientists, medical specialists, etc.

This whole group of users is held within defined age limits, somewhere between 15 and 65, but still includes many people with special needs, particularly in the context of this study, such as elderly and disabled.

4.1.2 The leisure population

Other people have requirements for interaction with public services utilising IT, such as information retrieval services, shopping and banking services and even self-service vending machines for many everyday activities, such as car parking, purchase of tickets and general small items of merchandise. This group is characterized by a very much wider range of age, possibly from as young as five years, up to 80 or even 90. A large proportion of this group may have special needs, not just visual and physical handicaps, but mental handicaps as well, and simply by being very young or very old. The range of capabilities and aptitudes is large and includes both those with no education or training, and users with highly specialized training and education and a wealth of experience.

This leisure group has at least as great a requirement for language and cultural exchange as working people, if not more so. However, in general their encounters with texts in other languages are more sporadic and they have less familiarity with IT. Another reason for highlighting this group is the proportion temporarily or permanently embedded within a culture other than their own, with the requirement, therefore, to be able to communicate to a culture outside the one in which they dwell.

4.1.3 People with disabilities and the elderly

The significance of visual deficiencies lies especially with those having acuity or astigmatism problems, who may have a problem perceiving fine detail in character sets, making it especially difficult to distinguish diacritical marks or the detail of some non-Latin characters. The general significance relates to form and fonts and character size, over all of which the user should be able to have some control.

Blind IT users are now beginning to be able to master graphical user interfaces supported by combined sound and tactile interfaces, after a period with great problems introduced by new, and for them, inaccessible technology. When introducing new CST these users expect not to be left behind again, but to find their alternative ways of displaying and controlling information built into the user interfaces from the start. One such need is the alternative representation of characters in Braille for tactile display.

Physical deficiencies are an issue relevant to this study, because of their impact on manipulation of IT input devices. There are also more general issues of control and display devices e.g. text phones, Teletext (Minitel/Viewdata/Videotext), Braille (including keyboards), touch screens, mice and operating systems, not addressed in this study, but which falls into the domain of ETSI.

Nationally developed systems are in a critical situation now in the area of internationalization. One such is text telephony, mainly used for text conversation in the telephone network by deaf, hard-of-hearing, speech-impaired and deaf-blind people. The users are impatient to get support for an internationally useful replacement of the currently incompatible national systems.

Teletext is the medium used, for example, for subtitling of TV. It has a general requirement for the internationalization of character representation, but also for standardising use of colours and other effects of expressing specific non-language related contents of the TV programme.

Finally, support for sign languages within the context of CST should be developed to cater for the need from transcribed documentation and education in these languages. The symbolic languages used by people with language disabilities i.e. Bliss and Pictogram needs support in the international IT environment both for local and remote communication. Education in pronunciation is highly related to modern IT implementations, and a support of phonetic transcription is needed in that sector.

An important long-running initiative within the European Union is the so-called TIDE programme (Telematics for Disabled and Elderly People). Since 1992 TIDE has been running with the aim to improve the quality of life for disabled and elderly people and to improve the European industry and market in products and services that meet their needs.

Past and current TIDE projects and other activities in the area are summarized below:

Braille: progress on standardization, as Braille alphabets are not all identical, e.g. work in ISO TC 173, work on contraction systems, text to Braille conversion.

BLISS: which is a picture symbol language for use by severely motor and speech disabled, employing a standard Bliss-symbol Graphical Character Set (Reg. no. 169 in the ISO International Register of Coded Character Sets to be used with Escape Sequences).

Sign language representation: many deaf people communicate with sign language, which needs coded representation. TIDE has three projects dealing with signs.

Text telephones: used by many deaf, hearing impaired, speech impaired and deaf-blind people. Activities in COST 219 and COST 220, and standardization work in ETSI and ITU.

Teletext: international exchange of television programs, including Teletext, raises the problem of conversion and transliteration. A similar problem arises with sub-titling.

Character sets and control codes: need for definition of minimal subsets of existing standards, e.g. ISO/IEC 10646-1, ISO 6429. In particular real-time control of conversation is critical.

Symbols for signing and posting: is used, for example as a standardized way of informing which accessibility features for disabled people are available in an application. Work to define more symbols is ongoing in ETSI and ITU.

4.1.4 Intermediate users

These needs are specialized and professional, and they originate from users such as standardizers, political administration, procurers, manufacturers, application developers, service providers, user organizations and professionals such as librarians and linguists. These people normally also have the role of end users -- using the technology for their job and leisure -- but has the added role of being responsible for others' use of the technology too.

The requirements of intermediate users originate from the intrinsics of the technology, that is how it works, and not just the results which are the main interest of end users. Many of the requirements on how it works are essentially the same as end user requirements, such as functionality, consistency, inter-operability, preciseness, efficiency, security, economic viability.

But some standardization requirements come from the question on how to make it work, such as guides for making standards in the field, guides for implementation in applications, standards for APIs to be used by application developers, generic standards on APIs for programming language standards developers, standardized profiles for which coded character sets to use in communications, registry standards for reference in other standards, and overall description of all existing and required standardization efforts with priorities and work allocation provided. Actually almost all of the standards are written for the use of intermediate users, as end users do not care about for example how characters are coded or applications are programmed or letters are communicated, as long as it works.

A summary of requirements in the field of CST for different intermediate types of users is given below. The individual requirements are further elaborated and listed in clauses 6 and 7.

Standardizers have major requirements on a diversity of levels. First of all they use each others' standards, for example the coded character set standards are used in programming languages, and the programming languages are used to define database languages. For this use to be consistent for example across programming languages or communication standards there is a need for guides on the use of CST in such standards. Member bodies have requirements for their national and cultural specifications to be precisely defined and referencable uniquely, and standards writers have a need to be able to use this information in a well defined way. Last, but not least, standardizers need to know how all the standards work together in an appropriate model, so they know what is available or expected and plan accordingly.

Political administrators such as the European Commission or national governments require that the CST requirements for the use of their citizens are fulfilled. They need a standards apparatus to ensure that this can be achieved in a non-monopolistic market in a economically viable and inter-operable way. They also require overviews of the technology so they can fund what is necessary to ensure the fulfilment of the citizens' requirements.

Procurers, manufacturers, application developers, and service providers require the standards and possibly guides of using these, and also information on available standards and new work.

User organizations and professionals require standards to fulfil the requirements of the people they work for. For them, overviews of needed standards and guides are essential to avoid gaps.

Many standardization efforts will thus need to satisfy a number of user audiences and the diverse audiences may make it more difficult to reach consensus.

4.2 Technical aspects

4.2.1 General

The first basic task is reading. Clearly many other applications, if not all, involve this. Thus, information retrieval of all kinds, from public information sources such as signs and notices, transport departure and arrival boards, timetables, electronic data bases and news sources, teletext, etc., all require perception and above all comprehension of the displayed text images.

The second basic task is writing. Interactive tasks, where users have a requirement to input characters from a keyboard, offer an additional complication, and users must be offered a familiar character set, which they must be able to recognise, or the instructions on how to enable it. User interface design is outside the scope of this work and much design guidance material is available, including that on characters.

Character encoding has evolved to facilitate transliteration, or the representation of spelling in characters of another language, and conversion. With time this has satisfied more and more of the users needs, but the field is constantly expanding. Many coding schemes have been employed and their requirements are discussed in clause 7.

4.2.2 Text and data processing and communication

Transliteration and transcription are used when the letters of the original text are unavailable or incomprehensible for most readers. In bibliographical or other scholarly work the need for an exact representation of the original text calls for transliteration, while transcription is used to write e.g. Greek and Russian names in books and newspapers for the common reader. It may also be necessary to incorporate fallback representation to avoid annoying errors when trying to write certain letters where adequate input/output facilities are not available. In addition, a user needs to be able to receive and convey information that may be in a convention or notation peculiar to a particular nationality or culture (including business or scientific sub-cultures).

The following are areas where different notations may be used, which may be culturally dependent (a more comprehensive treatment is given in [5]):

personal names
inflection, hyphenation and spelling
numbers, ordinals and measures
monetary amounts
date and time
coding of national entities (currency, etc.)
telephone numbers, mail addresses
payment account numbers

In all applications, messages to the user and their expected responses must be capable of being selected in language of choice. This aspect is discussed further in clause 6.

The number of characters in the Basic Multilingual Plane (BMP) of the Universal Character Set is 65.000 characters, but with the addition of further planes the repertoire is expected to become up to 250.000 characters. Users will expect to be able to enter and see any or all of these characters with their equipment. This aspect is discussed further in clause 7.

4.3 Conclusions

4.3.1 Summary of task and user requirements

• use IT in a language and script of their choice, both for input and output,
  
• work or play in their own national or cultural environment and
  
• interact with other cultural environments.

To facilitate communication and interaction outside the boundaries of their own culture or nation, many elements of these requirements need a degree of standardization. This is addressed in later clauses of this report. Many individual and national or cultural requirements have already been identified, and some references to standards in this area are attached in Annex D.

4.3.2 Prioritization of user requirements

• 4A1: Applications should allow users to communicate and write the full orthography of their own language and other languages of choice, including symbolic languages;
  
• 4A2: Applications should allow users to select a variety of scripts and character repertoires in a variety of fonts and sizes;
  
• 4A3: Applications should be able to accommodate the requirements of elderly and disabled people with respect to both input and output devices;
  
• 4A4: A specified set of characters and a repertoire of telephone control functions and codes should be specified for text telephony as a contribution to the current standardization in the area;
  
• 4A5: Character sets and control code repertoires should be developed to cater for the needs of disabled people;
  
• 4A6: Systems with inadequate code tables should provide fallback representation;
  
• 4A7: Applications should prompt users for the appropriate orthography/notation (with regard to cultural conventions) for the culture in which a user may be working;
  
• 4A8: Applications should contain messages and help in a language of choice, and be capable of receiving the same language.
  
• 4A9: All public IT services, especially interactive services, should allow selection of language of choice;

Medium priority requirements:

• 4B2: BLISS or similar symbol language in CST should be included to cater for the severely motor or speech disabled;
  
• 4B3: Teletext should use the same character set technology as the rest of IT;
  
• 4B4: Transliteration possibilities should be available in all applications;
  
• 4B5: Provision should always be available for people having special needs;
  
• 4B6: Symbols should be available to users in addition to the alphabetical characters of natural languages required.
  
• 4B7: Applications should allow users to generate and see all the characters of the Universal Character Set
  
• 4B8: Applications should allow users to request the identity of any character from the Universal Character Set.
  
• 4B9: A comprehensive user manual should be available in the language of choice which will explain the purpose and use of all characters in UCS, beginning with the (proposed) Extended European Subset.

5. The European context

5.1 The multilingual environment

The development of education, business, communication and leisure together with many other factors has led to the situation where most Europeans are able to understand - in many cases also make themselves understood in -- more than one language.

Europe today thus finds itself in a unique situation in the developed world. In a relatively small, relatively densely populated area many different languages and cultures are mixed. The different languages are used in swiftly growing intercommunication between different peoples and areas, something which puts obvious demands on the means of communication.

5.2 A growing need

5.2.1 Needs

The recent and ongoing upsurge in the development and use of multimedia is a striking example of culture-dependent, software-based communication. The need for localization of such software is an obvious one; the need for internationalization and localization of means of communication for professional purposes is of older date and becomes more urgent by the day.

5.2.2 Timeliness

As of 1 January 1995, the European Union has three new members, bringing in two new Union languages plus the Sámi languages into this very close co-operation.

Right now, discussions on the inclusion of the former Eastern Block European countries are progressing apace, and even if in all likelihood it will take 5-10 years before even the first extension of the Union is made in that direction, already business contacts are growing fast and administrative co-operation is being discussed.

It is thus clear that from the point of view of closer co-operation, the time is very ripe to start work on CST standardization for European needs.

b. Falling apart

In the perspective of cultural disintegration which already in several Eastern European countries has led to conflict and even civil war, it can hardly be doubted that better text communication facilities between the different cultural groups can only lead to better understanding and more concord.

5.2.3 EU initiatives

One of the most ambitious projects, the European Nervous System (ENS) -- also known as "Support for the Establishment of Trans-European Networks between Administrations" - is currently encountering difficulties of practical and financial kinds, and national implementations seem uncertain.

Related to ENS are the recently made Council Decisions on "a series of guidelines for trans-European data communications networks between administrations" and on a Community programme to "support the implementation of trans-European networks for the interchange of data between administrations (IDA)". These projects seem to be relatively firmly based in all relevant authorities.

Already in use is the Co-operation in Automation of Data and Documentation for Imports/Exports and the Management and Financial Control of the Agricultural Market (CADDIA). There are pilot projects running in some Member States; others are being installed.

Then there is the development in the EDI area, driven primarily by the private markets but with substantial support by the EU in the form of the TEDIS (Trade EDI Systems) programme, now in its seventh year.

Also, of course, the Commission is perhaps the world's biggest user of translation services and as such has far-reaching interests in tools which makes possible text transpositions from one culture/locale to another.

Last but not least one should mention the EC White Book proposals in 1993 for substantial investments in a European infrastructure for information and the EC Action plan for Europe's way to the information society in 1994.

Preservation and promotion of cultural and linguistic diversity is one of the guiding principles on which EU policy for the information society rests. The Commission (XIII-E and III-F) will prepare in May 1995 a Communication addressing European linguistic issues and means to stimulate the emerging language-based industry

This Communication will address the stimulation, coordination and regulatory initiatives to be undertaken in co-operation with the member states for the creation of a linguistic infrastructure of resources and services that improve language communication. It will also overcome language barriers hampering the development of the information society. The measures proposed will increase the use and efficiency of information and communication systems while contributing to the enrichment of the linguistic diversity of Europe and reinforcing Europe's language industry. [2].

The conclusion is that the European Union is making large efforts -- and planning perhaps even larger ones -- at creating a basis for an extensive, multi-purpose data communications network encompassing all Member States and reaching towards prospective members. Clearly these efforts will be seriously hampered if facilities for the handling of alphabetical and cultural differences are not in place.

5.2.4 EPHOS

5.3 European research

In 1994, three preparatory actions on reusable language resources were launched within the LRE-programme. They are POINTER on terminology data, SPEECHDAT on spoken language resources and PAROLE on harmonized textual and lexical resources and tools. These actions are expected to lead to standards proposals in this field that the EU may wish to develop into more formal European or international standards.

There are also other programs such as EUROTRA, TELELANG, TRANSTERM, GENELEX; EUROLANG and GRAAL that work on the linguistic problems within IT.

However, it should be pointed out that the majority of the projects are not concerned with character sets issues, the focus lying on the linguistic aspects. Many European languages can be written and presented by use of one single 8-bit code table, and for that reason the character sets issue is treated as lower layer problem.

There is also some privately funded research and development aimed at providing multi-lingual services in SQL data bases in more than 50 languages simultaneously and in a heterogenous environment. The results of this work are encouraging and deserve more support since both aspects of the problem are addressed: the interchange of data coded in different character sets and the linguistic aspects of such data and information.

Practically oriented work is being done both nationally and internationally within the European standardization framework. The CEN work described in Part 0 is one example.

In general, for reasons mainly to do with the multilingual European scene described in 5.1 above, research is being carried out in many places in Europe which concerns different aspects of CST.

5.4 The global connection

the topic is not relevant to other parts of the world, or

for whatever reason, global standardization will not be done in the foreseeable future.

In the second case, the European results will always be used as input into ISO/IEC if and when a corresponding project is started there.

Although Internet has made some progress on internationalization support in telecommunication standards, the same is not true for ISO/IEC and ITU-T. Work on internationalization for OSI purposes is done only in Directory and Virtual Terminal contexts. Basic work has been done within the framework of POSIX standardization, but there are problems with referencing that work outside POSIX standards. And while a basis for multilingual character set coding now exists in the form of ISO/IEC 10646 (see clause 7), much more pressure is needed in order to reach the desired extent of its implementation.

In Europe, even if the EU Member States do not always properly perceive the needs for CST standardization, there is a forceful standardization community and there is also the strong push and support by the Commission. The latter includes support for investigating the actual user needs.

5.5 Conclusions

5.5.1 Start in Europe

Some European standardization work in this area has already been started, such as the project of standardising a register of cultural specifications. This fact, plus the prominent position enjoyed by European researchers and technicians in the field of CST, is a good argument for Europe taking the lead in standardization. That way, Europe should be able to sustain (and build on) its current competence as well as pull other regions along -- even if the languages and cultural factors are locally dependent, the methodology is universally applicable.

It has been suggested that European CST standardization could include a pilot project -- perhaps a reference implementation -- exploring the practical use of ISO/IEC 10646 and also providing material for implementation guides.

It was furthermore stressed by the CST workshop (see clause 1) that close co-operation between R-and-D in this field and the standardization efforts is required.

5.5.2 Language priorities

A strong argument for not leaving non-indigenous languages outside the scope of CST standardization, however, is the fact that there are large groups in the EU speaking Arabic, Hindi, etc., and that those people by law have the right not to be discriminated against. Hence, the standardization bodies (and the Commission) must be prepared to argue its case if a decision as proposed above is made and the matter is subsequently brought to the Court of Justice.

Another argument for taking in non-European languages would be the matter of trade and business: it makes good sense from an economical viewpoint to facilitate business relationships by providing for localization.

However, it should be noted that standards and implementations for many "immigrant" languages and scripts are presently being developed in the home countries and regions of origin (Arabic, Indic, and East-Asian for instance). Standards bodies like CEN/TC304 should liaise with the corresponding members in those areas to facilitate provision of the necessary tools. For the present, it should remain a European priority to accommodate Europe's indigenous languages, however large or small, since no one else will be able to do so.

5.5.3 General European requirements

• 5A1: Europe should take the lead in the standardization work towards internationalization of IT for European languages;
  
• 5A2: Provision of support for the indigenous languages of Europe is a task for Europeans;
  
• 5A3: National bodies should be involved in this work;

Medium priority requirements

• 5B1: Pilot projects for implementation of new technology for internationalization should be encouraged;
  
• 5B2: European projects should liaise with projects in the rest of the world, to facilitate provision of support for languages other than the indigenous languages of Europe;
  
• 5B3: Standardization projects and R-and-D projects should work closely together;

6. Internationalization

6.1 General introduction

Users want applications that are truly adapted to their cultural environment, so that the use of the application feels completely natural with respect to the user's cultural expectations.

6.2 Man-machine dialogue

Some language-dependent messages and input are commonly used, for example asking for confirmation or rejection (yes/no/cancel?), and may be standardized.

A vast majority of messages, however, will be defined by the specific application. The best solution here would be to help applications administer messages/input according to the different languages and cultures, and to give users mechanisms to supply their own input to the application. A standard for users (or national distributors or user consultants) to specify input and messages for a given application would advance the availability of customized man-machine dialogue significantly.

As guidance for the use of a consistent terminology in a language, a recommended list of IT-related terms in each language could prove very useful.

In essence the user requires that the man-machine dialogue is in a language that he can understand, normally his own language. If the dialogue is available only in other languages, then it is a requirement that a natural language translator be present. This will also facilitate communication between people not having a common communication language.

To have all applications issuing messages and accepting input in all languages of the world seems to be a large problem and a difficult task.

6.3 Coded character set support

6.4 Specific cultural issues

date and time formatting and input
numeric input and formatting
sorting of text strings
man-machine dialogue
input and output of messages in the native language
paper formats
telephone number formats
postal addresses
hyphenation
spell checking
recommended use of standards
necessary characters to be processed
input methods
transcription rules
terminology

Machine translation services give an additional dimension to these problems.

6.5 Current application support

6.5.1 POSIX coverage

6.5.2 Other work

6.6 POSIX as a model for internationalization

6.7 Remaining user requirements

To satisfy fully the user requirements in all applications may take a long time. It is desirable that the user gets the best possible service in the meantime. This may be accomplished by specifying preferences for alternatives, for example messages in another language when support for the preferred language is not available. The other example is the provision of an alternative representation if the equipment available is not capable of correctly presenting the characters. The internationalization model needs to be enhanced to provide fallback representation, and APIs need to be specified for the fallback support.

6.7.1 Additional cultural information

6.7.2 Cultural register

Another problem is that, for a number of cultures, data are not available. Reliable data are hard to obtain. A specified process would help. CEN/TC304 is currently working on a standard for registration of such data. National work on collecting and obtaining consensus on cultural data should be encouraged.

When reliable data are available, they should be uniquely identifiable world-wide and easily accessible. Then any software producer could get hold of the information, and every user would be able to know and specify precisely what behaviour is wanted. The data should be available in a formally specified form and electronically, so that applications could process them without change (possibly via operating system services) and automatically deliver the desired support. The forthcoming CEN Cultural Register Standard provides a means for this.

For the use of the POSIX locales and charmaps, reference to the Cultural Register should be built into the standards concerned; examples include:

the programming languages C, C++, Fortran, Cobol, Ada, PL1, Basic, APL, LISP, Modula, Pascal,
communication standards like MOTIS (X.400),
the Directory (X.500),
database languages like SQL
operating systems standards like POSIX.

This should be done at the ISO level, and it is therefore desirable that the forthcoming CEN cultural register standard be further developed into an ISO/IEC standard.

6.7.3 Support in programming language and other standards

Such guidance for the design of programming languages is the subject of a revision of ISO/IEC TR 10176. Guidelines for APIs for cultural conventions and Programming language independent APIs are the subjects of ISO internationalization projects in ISO/IEC JTC1/SC22/WG20.

6.8 Specification technique guidance

6.8.1 Guidance to member bodies on specification of cultural conventions

6.8.2 European locale

6.8.3 Ordering of large repertoires

6.9 Promotion

6.10 Conclusions

• 6A1: Industry needs to get reliable information on cultural conventions in Europe;
  
• 6A2: National bodies should be encouraged to collect national data on cultural conventions related to IT;
  
• 6A3: Registration procedures for cultural conventions is urgently needed;
  
• 6A4: Guidelines should be written for national bodies on how to specify cultural conventions for registration;

• 6B1: Formal specification techniques should be extended to cover more types of cultural conventions;
  
• 6B2: The common European cultural conventions should be documented where possible and provided in a European POSIX locale, to provide a common ground to specify the more local conventions;
  
• 6B3: The available cultural data should be promoted to users, industry and service providers;
  
• 6B4: Reference to the Cultural Register should be built into the relevant IT-standards, like programming languages, communications standards, directory, database and operating systems;
  
• 6B5: European registration procedures should be developed into international ones;
  
• 6B6: Parallel IT terminology in natural language should be developed;
  
• 6B7: The POSIX internationalization model should be enhanced to provide multiple fallback preferences in case of missing data;
  
• 6B8: A standard of APIs for handling messages, provided independently of the program, should be developed;
  
• 6B9: A central process should be started to collect cultural data from National Bodies;
  
• 6B10: Guidelines should be written for programming language standardizers on how to design internationalization functionality in programming languages;
  
• 6B12: A standard for machine translation of natural language should be developed;
  
• 6B13: A standard for specifying cultural data independent of POSIX should be developed;
  
• 6B14: IT standards, like programming languages, communication standards, database and operating systems, should be enhanced to support internationalization functionality;
  
• 6B15: A language independent API specification for internationalization should be developed; default tailorable collation of the repertoire of ISO/IEC 10646 should be developed;
  
• 6B16: Promotion should be done of cultural standards and specifications;
  
• 6B17: A European sorting standard for European characters should be developed.

7. Character sets

7.1 The evolution

A short list of coded character sets currently in use includes the 7-bit ISO/IEC 646, its predecessor ASCII, other national variants of ISO/IEC 646, the 8-bit ISO 8859 series (regional sets covering e.g. Eastern and Western European languages, Greek and Arabic), ISO 6937 (covering the Latin script in an 8/16 bit code), the Japanese, Chinese and Korean 14-bit sets (JIS X0208 and X0212, GB 2312 and KSC 5601, respectively).

The only standardized coded character set which is intended to cover all languages of the world is the recent ISO/IEC 10646, also known as the Universal Multiple-Octet Coded Character Set, or UCS. In order to satisfy that purpose, however, it is necessary to use different codes and different levels of support. The standard is still being extended, but with some 33.000 characters it covers most of the languages in use today.

There are also a myriad of non-standard character sets developed by manufacturers to cover the same needs. Notably IBM has been creative, with a family of EBCDIC character sets (foremost among those incompatible with ISO/IEC 646) and PC code pages.

The X.400 applications supports the 8-bit coding systems defined by ISO 6937 and any set specified in the ISO/IEC character set registry. The X.500 Directory Service uses the T.61 (ISO 6937) coding, which caters for almost all characters used by Latin alphabet languages.

Recently, new objects have been introduced in the Directory Service which make internationalization and language specification possible, but as yet no applications with these new features exist. Since June 1992, the Internet also has the means to exchange messages containing multiple character sets through the methods defined in the specification of MIME (Multiple-purpose Internet Mail Extension). The World Wide Web service now has started to implement the underlying protocol HTTP, based on 8-bit character coding.

7.2 Towards the ideal situation

Nevertheless, the user ideally should perceive that the character set interface of his choice is the same all over the world, regardless of region or country where the data originated, and without intrusion of any underlying technical complexities of communication service or application program.

There is therefore a major need for conversions between all existing coded character sets. UCS is the primary building block for this, since it will encompass all the other character sets. Its implementation will provide the required integrity of characters as well as the required support for multi-linguality in the advanced network services.

7.3 The current situation

Thus despite some promising signs, Europe and the rest of the world are still of mixed opinions regarding the use of character sets. The users still use equipment based on 8-bit coding, the telematic services do not widely implement UCS, and the old standards are still implemented.

Some of the major reasons for the lack of UCS implementations are the following:

• Many of the existing applications still running can support only character sets used in their systems (mainly 8-bit based proprietary codes).
  
• Some of the widely used protocols over the international networks are hostile to 8-bit coded data and only let 7-bit data go through uncorrupted.
  
• There is still no mechanism for easy and default recognition of a stream of data coded according to the UCS scheme.
  The users believe that the 8-bit character sets will be present for long time yet in Europe.

Despite this, one property of the existing IT applications is encouraging, namely that some of the current and most important telematic services have the capability to transport a large number of different character sets, including UCS.

7.4 Discussion of issues

7.4.1 UCS provision

a) Input

It is expected that input in the future be generated on keyboards not very different from current keyboards, which have about 100 individual keys. There is a user requirement for standards for generating UCS characters with only a limited set of keyboard keys. These standards are currently varying from culture to culture, and standardization should cater for this cultural variance, while also offering one or more globally standardized input methods for UCS. The keyboard WG of ISO (ISO/IEC SC18/WG9) has recently started a new project assigned on the issue of global input method standards, while the CEN cultural registration standard can cater for registering culturally dependent input methods.

b) Processing, communication and storage

The different encoding forms and levels of UCS have different properties and also requirements for support by the application. Applications are usually written in a programming language and require the support by an operating system, and it is thus essential for proper programming and portability of applications that programming languages and operating systems support the various encoding forms and levels of UCS. The ISO group on programming languages, operating systems and programming environments (ISO/IEC JTC1/SC22) has several projects that address the issue of UCS support in the operating system, including POSIX, an internationalization framework technical report, a guide on the design of programming languages and language independent APIs for internationalization.

Communications standards need to be upgraded to be able to handle the different forms and levels of UCS, in a similar way to the operating systems and programming languages described above.

As the different encodings and levels of UCS have different capabilities and strong and weak points, UCS can be expected for as long as we can see to be present in different forms, and a requirement to be able to handle conversion between these coding forms is a major one. Specifications of these conversions are included in the standard, but the capabilities need to be present in applications to prevent users from having problems. Software for this conversion between UCS encodings is freely available. A related issue is the conversion to and from smaller character sets, which is covered in clause 7.4.2.

Getting applications to support UCS is thus a major goal. This cannot be controlled by standards authorities or public authorities, but progress can be helped by public procurement requirements and publicly funded programmes for development and awareness.

c) Output

There are a number of output requirements for the support of UCS:

• Equipment capability and fallback rendition
  

  Equipment, including printers and display devices, should be capable of displaying all UCS characters and need to handle UCS combined characters correctly.
  

  The requirement of full output support can be expected to not be fulfilled in many cases. The availability of glyphs and fonts may be quite limited due to hardware limitations etc., and the evolution of UCS with additions of new characters may render equipment out-of-date. The definition of European subsets of UCS will solve this problem for most equipment, assuming that fonts for the subset(s) will be made available in the public domain.
  

  For older and less advanced equipment, rules are needed for fallback rendition of non-displayable characters. Such rules should be designed in a forward-compatible (future-enabling) way.

• Transliteration
  

  Most users of IT would not be capable of precisely identifying or understanding all UCS characters (estimated to exceed 250.000), and ways to convey the meaning of characters need to be present. Many users would only be able to recognize a limited number of scripts and other characters need then to be transliterated to the relevant scripts to be understood by the reader. This transliteration may be culturally dependent (= transcription), and there may also be global transliteration specifications. ISO has produced a few standards, mainly for bibliographic purposes. CEN/TC304 has started work on transliteration, but further work needs to be done on how to specify cultural transcription and fallback rules.

• Character information help services
  

  Another requirement is to exactly identify a character, and possibly get information on where, how, and why it is used. The ISO/IEC JTC1/SC2/WG3 character set group is currently working on establishing a database with such information, and the ISO/IEC JTC1/SC18/WG9 keyboard group is also working on specifying the user interface related to this issue. Further procedures for formulating queries on such information could be specified, maybe in conjunction with the above fallback and transliteration services.
  

  Even if such a database is set up, reliable information on European characters must be collected by Europeans. To ensure the quality of this information, sponsored European research work is needed.

d) Subsets of UCS

Some of the implementation problems discussed above could be solved by the provision of subsets of UCS. CEN/TC304 has already started work on defining European subsets, particularly aimed at solving the problems of outputting the full character set of UCS.

It is estimated that full UCS implementation will be costly in the first stages of UCS use, and that manufacturers then only will implement a subset. To ensure that a common subset which can be used by the vast majority of European users be available for a reasonable price, and as a guide to manufacturers, it will be helpful to users and procurers of systems if European subsets of UCS, encompassing all characters for use in European languages plus other frequently used characters, be specified. Also such subsets may be useful to do further standardization work for example on sorting, so that the work is reasonably limited and still useful in an European environment. Only a small number of subsets should be specified. CEN is currently working on such European subsets of UCS.

7.4.2 Coexistence issues

Fallback methods

Inherent in the specification of character set conversion is how to handle inconvertible characters, including information preserving and information losing fallback techniques, and as different requirements may be present, it can be expected that different solutions be needed. One requirement is information preserving fallback, where the fallback representation would be legible to the user in the character set available, and where fallback representation is only done when needed, thus not disabling the user further from the limited character set. Another requirement is the preservation of the number of characters, but possibly information-losing.

This work is being investigated by different groups, e.g. CEN/TC304/WG4, IETF and TERENA. All this work needs to be brought together in a single set of specifications.

7.4.3 Coexistence

7.4.4 Other coded character sets.

7.5 Conclusions

• 7A1: There is a need to handle any European character
  
• 7A2: ISO/IEC 10646-1 at level 1 should be implemented in systems
  
• 7A3: It should be possible to input European characters via standard or extended keyboard
  
• 7A4: Programming languages and operating systems standards should support the encoding forms and levels of ISO/IEC 10646-1
  
• 7A5: Communication standards should be updated to support ISO/IEC 10646-1
  
• 7A6: A conversion capability between different forms of ISO/IEC 10646-1 is needed, as a minimum between UCS-2, UCS-4 and UTF-8
  
• 7A7: Peripherals such as printers and display devices need to be able to output the European ISO/IEC 10646-1-characters
  
• 7A8: Definition of European subsets of ISO/IEC 10646-1 is required to provide a minimum capability to European users
  
• 7A10: Provision needs to be made for the conversion between ISO/IEC 10646-1 and other coded character sets
  
• 7A11: Public awareness programmes are needed to promote ISO/IEC 10646-1

• 7B1a: Guidelines are needed for the design of programming languages in relation to UCS
  
• 7B1b: Guidelines are needed for the design of language-independent APIs in relation to UCS
  
• 7B2: Public procurement and research and development programmes should encourage applications to support ISO/IEC 10646-1
  
• 7B4: There is a need for fallback representation of non-displayable characters; there is a need to provide forward-compatibility for characters as yet undefined.
  
• 7B5: Display systems should enable users to identify uniquely a displayed character
  
• 7B6: Registration of cultural conventions in the CEN register should be encouraged
  
• 7B8: A specification for fallback treatment of inconvertible characters should be developed
  
• 7B9: It needs to be ensured that all groups working on code conversion should co-operate to produce a specifications which are compatible.
  
• 7B10: Sponsored European research work is needed to collect reliable information on the history and use of European characters
  
• 7B11: Guidelines are needed on the use of character sets in Europe
  
• 7B12: Bar code standard for European use is needed
  
• 7B13: Enhanced OCR-B standard is needed for European use

• 7C1: There is a need to handle any character world-wide
  
• 7C2: It should be possible to input any character via standard or extended keyboard
  
• 7C3: Peripherals such as printers and display devices need to be able to output all ISO/IEC 10646-1-characters

8. Implications of the ICT workshop recommendations

8.1 Organizational structure

The linguistic aspects can easily get lost in structures such as Fora, Consortia and ad-hoc developments of publicly available specifications. This kind of structure may counteract a true multilingual infrastructure in Europe.

The single workshop approach is not in itself a threat to a true multilingual infrastructure in Europe, as long as the proper management tools are in place.

8.2 Future role of national standardization organizations

For minority cultural groups, national bodies may not be the best channel to get heard internationally, so other means must be found as well, which do not exist today.

8.3 International standardization

8.4 Other issues

8.5 CST-specific organizational requirements

• 8A1: Any European organization working on IT-standardization should take due account of the linguistic and cultural aspects of the information society
  
• 8A2: National standards bodies have an important role concerning the linguistic and cultural aspects of IT-standardization
  
• 8A3: Europe should take the lead in the internationalization of IT

9. Summary of user requirements

9.1 The users

• to be able to communicate (with computers and via computer networks) in a natural language of his own choice, i.e.:
  
  • to be able to use the full orthography of that language, and
    
  • to be able to use a variety of scripts and characters as necessary.

This is valid also for communication with and among disabled people. For this case, there is need for support from other ways of input and output of common languages or specific languages suiting other ways of perception and production. Examples are the symbolic language BLISS and natural sign languages. For the industry to be able to supply products suiting these requirements, the CST standardization must include them in the common work.

9.2 Cultural conventions

Examples of such services and products are

• the X.500 Directory services
  
• World Wide Web services and related applications
  
• programming languages.

The industry needs coordinated registration procedures at the European level.

9.3 Internationalization and Europe

9.4 UCS

Solutions are also needed to related problems such as

• standardized methods for input and output capable of handling at least the European subset character sets,
  
• appropriate identification of the UCS characters.

Existing IT standards which include or presuppose the use of coded character sets need to be updated to cater to the use of UCS.

9.5 Coexistence

In addition, means for code transformation between different coding methods in user equipment and telematic services must be provided either in the user environment or in the services. The use of a European UCS subset must be taken into account.

A particular way of handling the coexistence problem is the fallback representation. Proper methods for transcription/transliteration also taking into account the needs of disabled and elderly people must be provided.

PART II -- A Taxonomy Of Character Set Technology

1 Introduction and scope

-a taxonomy helps to identify all aspects of the domain in question which might be subject to standardization;

-a taxonomy helps to provide a logical structure for the standardization activity.

A taxonomy has been developed of relevant concepts in the domain of character-set technology, based on user requirements for functionality, as discussed in Clause 4 of Part I of this report.

By way of an application, all known current standards and standardization activities have been grouped according to this taxonomy, thus forming another type of taxonomy, that of the standards themselves.

2 A taxonomy of character set technology.

The present classification of CST concepts was made through the identification of commonalties, such as characters, sets, fonts and rules relating to presentation. The analysis was based on a much wider view of "multi-cultural support", a shown in Figure 1, which attempts to map some of its concepts. Areas relevant to this report were chosen and developed into the full taxonomy, shown in clause 3.2. This latter choice comprises the technology which relates to methods for specifying, and rules governing, the creation of unique properties and codes which facilitate the presentation, storage and transmission of individual characters.

The taxonomy in clause 3.2 was based on references ISO/IEC TR 10000-1, ISO TR 12382 and IEC 824 and the activities of appropriate standardization bodies, but most notably the work of CEN/TC304 and ISO/IEC/JTC 1.

3 Description of classification

3.1 Description

The taxonomy in clause 3.2 takes the classic form of a tree structure, where two major classes are recognized; Locales and Characters. The former deals with the cultural environment of the user, the latter with the smallest divisible parts that make up the messages which are being electronically processed.

A taxonomy of whatever phenomena can be constructed in several ways, depending on its purpose and the aspects applied. (For instance, a number of persons may be grouped firstly according to age, then according to gender, then according to place of living -- or precisely the other way around, according to need.) A taxonomy for standardization purposes naturally has to take into account the most practical ways to group existing standards and standardization projects as well as the logical connections between them and any conceptual "holes" which may need to be filled in order to cover the full need for standardization.

The following taxonomy is thus intended to provide a map for almost all of the user requirements identified in Part I (see the application in Part III). Therefore the level of subordination in some cases go very deep -- this does not mean that the actual standardization projects need a taxonomy of the same complexity. When a sub-level is empty of existing or future standards, the entries in that sub-level are simply collapsed and only the level above remains.

3.2 Taxonomy for CST and internationalization

/ (no id) TAXONOMY OF CST AND INTERNATIONALIZATION

4 Taxonomy of current standardization work and research

5 Maintenance of the taxonomy

PART III -- Strategy for implementation

1. Introduction

2. Promotion/awareness

3. Research and Development

• basic research before standardization can take place (pre-standards work)
  
• the development of research work into standards
  
• the development of products using CST

Research and development related to CST standardization could be included in the EC Commission's IV Framework programme.

See also Part IV.

4. Standardization

The descriptions of work items and programmes in Part IV have been made with this in mind. However, constant vigil is necessary if the required awareness is to be built and maintained.

5. European and ISO/IEC standardization

6. Coordination

7. Role of particular organizations and standardization bodies

• CEN : Act as a motor for the production of European standards. Coordinate national efforts including promotion/awareness. Handle the contacts with the European Commission. The CEN administration, therefore, is essential to the success of all European standardization efforts, including the application of standards.
  
• CENELEC : Include multi-cultural (UCS) support in IT application standards.
  
• ECMA : See industry.
  
• ETSI : Include multi-cultural support in Telematic standards.
  
• EWOS : (TG-CS) Promote correct use of Character Set Technology standards within the expert groups in their development of standardized profiles and other standards for the Open Systems Environment and the Global Information Infrastructure.
  
• ISO/IEC : Include multi-cultural support in IT and IT application standards. In cases where ISO/IEC is doing this too slowly, the work has to be done by European organizations, applying the Vienna agreement.
  
• National standardization bodies : (members of CEN) Channel national input into CEN and ISO/IEC work. Handle contacts with authorities and other organizations including the promotion and awareness of the use of standards. In the CST context: organise input of reliable information for cultural conventions and elements.
  
• The European Commission : Coordination. Speed up the whole process and facilitate the work by partly funding the R-and-D work, standardization and promotion actions. The success of these actions will depend also on enforcement of the developed standards through regulations for procurement in the public sector. Organize specific promotion of applications, testing and certification. Possibly also introduce special education programmes.
  
• National governments : Transform EU legislation and recommendations (including the European Commission activities) into action on the national level. Supervise and promote multi-cultural IT applications. Promote awareness through education etc. Support standardization through funding and other means. Require and organize testing and certification.
  
• Industry : Provide expertise for standardization work. Use final results in products -- this is crucial! Participate in R-and-D. Provide testing and certification facilities. Develop own standards in Fora such as IEEE, ECMA, X/Open, and feed them into the formal standardization process.

PART IV -- Work needed for multi-cultural support in IT

1. Introduction and scope

Promotion, coordination, research and development activities, based on the discussion in Part III, are outlined in clauses 2 and 3.

In clause 4, the requirements in Part I are grouped according to the taxonomy in Part II. The items in clause 4 are then in clause 5 subdivided according to status: completed or ongoing work and new work to start now or later.

New work items are described in clauses 6 9. Each clause is related to the organization which is recognized as the one mainly responsible for the respective technical area.

Finally the financial aspects are discussed in clause 10.

2. Promotion and coordination activities

2.1 Promotion

• Press releases at major stages in the standards development, e.g. when a proposal is circulated for voting; when new projects are approved. This may be done at CEN/TC 304 level and via the member bodies.
  
• Information by electronic means. Networking becomes more and more important in the information society, and many IT professionals and users have network access. Examples:
  
  • The World Wide Web services should be employed to inform the general public. Suitable documents include press releases, general introduction material and specific documents related to the standards. (For copyright reasons, at present the standards themselves cannot be published this way.)
    
  • The Usenet netnews includes a number of newsgroups related to this field, e.g. comp.std.internat.
    
  • An electronic mailing list should be used for the communication of press releases and other relevant material. It could also be used for discussions with special interest parties. This list should preferably be a joint effort by standardization organizations concerned.

2.2 Coordination

Fora for the dissemination of information on standardization work and to obtain user input have proven useful. Regular workshops is another way of providing user interface.

R-and-D projects for the development of reference implementation is one way to increase the practicability of the standards. Support of such projects could be given over the normal R-and-D programmes in the EU.

3. Research and development activities

• Basic research needed before standardization can take place (pre-standardization)
  
• Developing R-and-D proposed standards into formal standards
  
• Implementing Character Set Technology in products

3.1 Required pre-standardization work

3.2 Developing R-and-D proposed standards into formal standards

• Terminology data exchange format (POINTER, TRANSTERM, GENELEX)
  
• Models on spoken language (SPEECHDAT)
  
• Harmonized textual and lexical resources and tools (PAROLE)
  
• Guidelines and procedures for development of internationalized software products (GLOSSASOFT)

CEN/TC304 is expected to develop such proposals into formal standards. This is reflected in the standardization programme below.

3.3 Development of products that implement UCS

The problem of character sets appears with all its dimensions when the products are put on the network and included in the telematic service. In telematic services, the minimum common denominator is still ISO 646 with some minor exceptions i.e. in electronic mail, where 8-bit character sets codes are used.

For this reason the scope of the R-and-D work in developing and implementing Character Sets Technology and Cultural Conventions for IT is in the CST taxonomy class C/3332 (Internationalization support in Telematics). This would provide multi-cultural support facilities in the telematic services which are now under development in Europe (ENS programme is just one of them).

The scope of the R-and-D work should be in specifying, implementing and developing:

• human-computer interfaces for telematic services which match the user's local environment, level of skill and language, taking into account various interface technologies such as multi-media, windowing and desktop facilities. This R-and-D task should include the needs of disabled and elderly people as well as the ergonomic aspects such as on-line help, tutorials and language translation tools accessible over the network.
  
• globalization of an inter-working client/server systems in heterogenous telematic environment for multi-lingual user community. Here, aspects to be addressed are interconnection of different type of data bases coded with different character sets, different data processing systems, HTML and SGML documents coded with different character sets (i.e. with KOI8 or ISO 8859/8 -- Greek) in WWW service etc.,
  
• general API for translation, conversion, servicing and rendering a wide range of used coded character sets with connections to different telematic services. The API should address the cultural conventions aspects of the user i.e. the translation and fall back procedures for unrecognisable or non-displayable characters must be user driven or be subject to user selection, and according to the registered European Locales,
  
• internationalization of the telematic services especially X.500, X.400 and WWW. Funding by EC relevant programmes and implementation in Pilot projects is required. Work within ISO is ongoing but is relatively slowly. Exact representation of legal bodies and personal names is a must for the Directory services. Characters in use in different countries should be supported. Attributes such as "businessCategory" must be represented according to the Locales provided by the service, suitable translation tools as a part of the service (or storage of multiple Locales) have to be available as well the API for conversion and transformation of coding,
  
• Pilot project in implementing ISO 10 646 in telematic application e.g. in the Directory. This task should be restricted in the beginning on the European Sub-repertoire. Provision of a framework for assessment of the benefits in using ISO 10 646 and possibilities for backward compatibility with existing system. Show the possibility of smooth transition.

4. Deriving the standardization action from the user requirements

5. Proposed European standardization programme

• Phase 1: Work items soon to be published by CEN or EWOS
  
• Phase 2: Other ongoing European work (in CEN and EWOS)
  
• Phase 3: New work items to start immediately (selection based on priority, maturity and dependency)
  
• Phase 4: Other new work items

The phase numbers are also included in clause 4 above, for cross-reference purposes.

An attempt has been made to identify, for each item, the organization responsible, related work, type of deliverable and a tentative time-table.

Already published results of European work are listed for information in clause 5.1.

5.1 Phase 0: Published European standards/reports

5.2 Phase 1: European standards/reports to be published soon

5.3 Phase 2: Other ongoing European work

5.4 Phase 3: New work to start immediately

5.5 Phase 4: Other new work

6. Description of new CEN work

6.1 CEN/TC304

Description: A taxonomy of approved and proposed standardization items will be produced and updated. In keeping with other IT standardization taxonomies, it should comprise classifier, title, description. responsible organization, current status and time-table.


Related: all.


Deliverable: CR/M-IT-. Plan: start 1996/7, FV 1998.

L/11113a Guidelines and procedures for development of internationalized software products. Phase: 3

Background: The Glossasoft R-and-D project aims to produce:
• guidelines on how to structure new software packages and systems
• methods and tools to help adapt existing packages and systems so that the natural language of interaction can be readily changed.
The second phase of this project will result in a first version of the guidelines and methods, focusing on the interactive elements in the interface necessary for the localization of existing software. Project deliverables are draft linguistic guidelines, draft software structure guidelines and draft method for localization of software.


Description: CEN/TC304 will take an immediate look at the Glossasoft deliverables to see how they can be develops into European standards.


Related: Glossasoft R-and-D results, L/11113b.


Deliverable: ENV. Plan: TC 1996-06, FV 1996-12.

L/11114 Harmonized textual and lexical resources and tools for automatic translation. Phase: 4.

Background: The Parole R-and-D project aims to define and contribute to the establishment of a collaborative infrastructure which will stimulate, throughout the EU, the creation and reuse of harmonized textual and lexical resources and tools. TC304 should be prepared to process proposals coming from this project.


Description: Process proposals for European standards coming from Parole.


Related: Parole.


Deliverable: EN-IS. Plan: start 1996/7, FV 1998.

L/1114 Models of spoken language. Phase: 4.

Background: Spoken language resources (SLR), for example annotated speech databases, pronunciation lexica and language models, are necessary both for present day speech driven applications and for the development of a technology capable of handling real, spontaneous speech. The goal of the SPEECHDAT project is to minimize the costs of SLR by creating an efficient infrastructure capable of ensuring the production of SLR for different purposes. Because of the sharing of resources, this project could develop a proposed standard method of describing human speech. CEN/TC304 should be ready to develop such a proposal in to a European standard.


Description: Develop a proposal expected from SPEECHDAT into a European standard. Difficult item.


Related: Speechdat R-and-D results.
Deliverable: ENV. Plan: start 1997, FV 1998.

L/1311 European default locale. Phase: 3.

Background: National cultural conventions are a major and important task to undertake for National Bodies. To provide some default for European National Bodies and also some guidelines, a European locale should be developed. National bodies can then just chose to concentrate on those aspects which are undeniably national and leaving the rest to follow the "European culture".


Description: The purpose of this work is to document those common European cultural conventions that exist, in the form of a locale, for registration under the registration standard on cultural conventions from TC304/WG2. The work is expected to cover at least European ordering rules and may also cover date format, transliteration and other conventions as found appropriate by the project.


Related: L/2111, making this a high-priority item.
Deliverable: ENV. Plan: TE 1996-09, FV 1997-06.

C/112132 General symbol language representation. Phase: 3

Description: A common notation system for sign language will be developed and standardized. The work will possibly have to be done in two steps, with an initial step consisting of a technical study of the existing notation systems, in order to find the set of features needed to give adequate support to all European natural languages.


Related: TIDE, JTC1.


Deliverable: ENV. Plan: TE 1996-12, FV 1997-06.

C/112211 Minimum set of control functions for Europe. Phase: 4

Background: ISO 2022 and ISO/IEC 10646-1 refer to control functions which are separate form graphic character repertoires. ISO/IEC 6429 specifies a complete set of control functions and their coding. ISO/IEC 10538 identifies a subset of the control functions of ISO/IEC 6429 for use in text communication. It also expands the semantic definitions of those control functions for use in the text communication application. There are many different methods for coding control functions. Control functions are classified by function (e.g. printing controls, media pointing controls, communication controls etc.). A minimum set of control functions is required for the operation of ISO 2022 code extension techniques (e.g. ESC, LS0, LSI, LS2, LS3, LS1R, LS2R, LS3R). Control functions are required to be used in conjunction with coded character sets being defined in projects of CEN/TC 304.


Description: It is proposed that an ENV be developed which defines a model for the USE of control functions, identifies a minimum set of control functions (BASIC) and identifies a full set of control functions for text handling (FULL TEXT). Consideration should be given to the possibility of progressing this specification outside Europe for adoption as an ISP since the document would be a profile of ISO/IEC 6424 and ISO/IEC 10538 and would complement the two ISPs being developed by EWOS.


Related: EWOS groups, SC2, SGFS, SC18, EPHOS Topic B, ECMA.
Deliverable: ENV (may be followed by EN-ISP). Plan: start 1996, FV 1998.

C/113a Character-glyph model for Europe. Phase: 3

Description: This work shall provide European input to and accelerate the international work started by JTC1/SC2 in Nov 1993 on Character/Glyph model.
Related: C/113b, JTC1.


Deliverable: ENV. Plan: TE 1996-12. FV 1997-06.

C/1132 Character/Glyph correspondence for Europe. Phase: 3

Description: Define which glyphs correspond to each character in the European subset(s) of UCS. European input to international work.


Related: C/113a+b.


Deliverable: ENV. Plan: TE 1996-12, FV 1997-06.

C/1213b General European rules for fallback representation. Phase: 3

Description: Specify multiple level fallback rules which would allow converters to the best solution available in the target character set. Work to be based on all current projects of TC304/WG4.


Related: C/1213a, C/1212, national work foreseen.


Deliverable: EN. Plan: PE 1996-10, FV 1997-10.

C/2111a Standardized profile of UCS keyboard for Europe. Phase: 3

Background: A work item has been assigned to ISO/IEC JTC1/SC18 to develop input methods to enter characters from the repertoire of ISO/IEC 10646. At the time of writing this document, a Committee draft (CD 14755) is the most current WI description. This includes a basic method using the hexadecimal representation of a character and another method involving selection of a character displayed on the screen. These methods are generic and apply to Europeans in the same way as to others. A third method allows composing characters by combining characters available on the user keyboard and, based on a system file containing mappings between such character combinations and the target character, have the desired character produced by the keyboard handler program. Since the engraved character repertoires of keyboards vary among countries, as does also the need to access foreign characters, there is a need for a technical study in Europe. The SC18 WI also includes a feedback method for identifying displayed (foreign) characters for later input. Again, this method is generic and needs no European additional work.


Description: A technical study to see if the combination method needs national adaptations, e.g. for the contents of the system file providing the mapping between key combinations and target character.


Related: CD 14755, ISO 9995, C/1213.


Deliverable: ENV. Plan: TE 1996-09, FV 1997-06.

C/313 Tools and transformation tables. Phase: 4

Description: Develop a technical report or ENV with description of methods of implementation of the General Model for Transformation (C/31311) and the use of the fall back rules and tables. The document will provide guidelines about how to act in different circumstances different character sets conversions and how to meet different requirements for transformation and conversion. It will explain the different types of conversions which can be used. It will provide reliable sources for conversion tables and implemented fall back rules on general (international) and national level (cultural dependent). Guidelines for use and implementation of the General model in different telematic services or conventional information systems with recommendations for relevant situations will be given. A list of available tools will be a part of the document.


Related: All C/313.


Deliverable: CR/ENV. Plan: start 1996, FV 1997.

6.2 Other CEN committees

6.2.1 CEN/TC224

6.2.2 CEN/TC225

Description: A bar code will be developed to cover the European character repertoires.


Related: prEN 1923 and ENV 1973.


Deliverable EN. Plan: start 1996, FV 1998.

6.2.3 CEN/TC251

Description: Include support for UCS and internationalization in standards for Medical Informatics, particularly systems dealing with personal data.


Related: EWOS-EG-MED, CENELEC TC62, prEN 1923, ENV 1973, prENV 12005, L/11113.


Deliverable: ENs/ENVs. Plan: start 1996, FV 1997/8.

6.2.4 CEN/TC278

6.2.5 CEN/TC298

6.3 CEN National member bodies

Description: Work for national bodies to translate standardized IT terminology (ISO 2382 series).


Related: ISO 2382 series.


Deliverable: national standards. Plan: various (not urgent).

L/2111 National standards on cultural conventions. Phase: 3

Background: National standardization bodies are the major originators of input for European Cultural Registry. Reliable information will be provided in the Registry if national standards are prepared and adopted for cultural conventions in particular country.


Description: Develop national standards on cultural conventions. Work to be performed by national bodies in co-operation with CEN/TC304.


Related: L/1211, L/212.


Deliverable: national standards. Plan: various (urgent).

6.4 Work in co-operation with ISO TCs

6.4.1 ISO TC10 (and CEN TC304)

Description: Amend standards for technical drawings and technical documentation. The specifications for text strings should allow any character of UCS or at least the European subsets thereof. Co-operate with JTC1/SC24, IEC TC3 +25 and ISO TC12 +145+130.


Related: SC24, IEC+ISO TCs, ISO/IEC 10646, ENV 1973.


Deliverable: EN-ISs. Plan: start 1996, FV various.

6.4.2 ISO TC37 (and CEN TC304)

L/11112a Terminological data exchange format. Phase: 4.

Background: Transterm addresses the problems of enriching terminologies and integrating them into the application dictionaries of National Language Processing (NLP) systems. It also deals with automatic and semi-automatic construction of application terminologies from corpora. The main objective is to facilitate the use of terminological data in NLP systems thus tackling the critical issue of real site customization of this type of software. The project co-operates closely with Eureka projects GRAAL and GELENEX.


Description: Develop Terminological exchange format (eventually data base model) in connection with the international TIF standardization in ISO/TC37 (ISO/DIS 12200, ISO/DIS 12620).


Related: Transterm R-and-D results. ISO DIS 12200 and 12620.


Deliverable: EN-IS. Plan: start 1996, FV 1998.

6.4.3 ISO TC46 (and CEN TC304)

C/3332p UCS in Library Informatics. Phase: 4

Description: Develop and amend standards for Library Informatics that support UCS and internationalization.


Related: EWOS-EG-LIB, ISO TC171, ENV 1973, prENV 12005.


Deliverable: EN-ISs. Plan: start 1996, FV various.

6.4.4 ISO TC173 (and CEN TC293)

C/1121112 Common 8-bit Braille character set. Phase: 3

Description: This WI defines a common 8-bit Braille international standard for data transfer. Correspondence and possible transformations between this and ordinary letter based coded character sets will be defined. Various contraction system to reduce volumes are presently in use, although often with limitations of use to certain nations. Control functions to indicate the contraction system used will be defined and standardized in order to facilitate inter-working between dissimilar systems.


Related: ISO TC173, JTC1.


Deliverable: EN-IS. Plan: TE 1996-12, FV 1997-06.

6.4.5 ISO TC204 (and CEN TC278)

Description: Standards for Traffic Telematics will be amended to support UCS.


Related: prEN 1923 and ENV 1973.


Deliverables EN-IS. Plan: start 1996, FV 1998.

6.5 Work in co-operation with ISO/IEC JTC1

6.5.1 JTC1/SC2 (and CEN TC304)

C/1111 Short identifiers for characters. Phase: 4

Background: Currently there is only one standardized and accepted character identifier scheme: the long English names of characters in UCS. This issue (short identifiers) was raized several times in SC2 but there were no resources to do the job. SGML provides something but it is not general. Character mnemonics are already used in some of the CEN work, and they are an example of short identifiers for characters.


Description: Develop a character identifier scheme using short identifiers.


Related: --.


Deliverable: EN-IS. Plan: start 1996/7, FV 1998.

C/1121b Update 10646 to include missing European characters. Phase: 4

Background: BMP of UCS does not cover all known characters. A revision of 10646 is expected to be published in about 5 years time, taking in to account the missing characters. A review has already started on which of the missing characters/scripts to include in the BMP itself. European historic scripts and individual characters from minority languages are missing in the current BMP, so input is needed from TC304.


Description: Revise 10646 to include the missing European characters, preferably in BMP.


Related: C/1121 a+c.


Deliverable: revized EN-ISO/IEC 10646. Plan: start 1995/6, FV 1999.

C/1121c Maintenance of ISO/IEC 10646. Phase: 3.

Description: Full implementation of the Vienna agreement, by parallel voting on all proposed amendments and corrigenda to the IS.


Related: ISO/IEC 10646.


Deliverable: EN-AMD/COR. Plan: on a stand-by basis.

C/113b Global character-glyph mode. Phase: 4

Description: This work item was allocated to JTC1/SC2 in Nov 1993. European input is expected from items C/113a and C/1132.


Related: C/113a (replaces ENV), C/1132.


Deliverable: EN-IS. Plan: start 1995, FV 1998.

C/114 Enhanced OCR-B standard for European use. Phase: 4.

Background: A proposal has come forward to extend this repertoire to cover all European Latin characters.


Description: If no revision is taking place in ISO then TC304 should consider undertaking the work because a number of important application standards are dependent upon OCR-B.


Related: ISO 1073-2, SC17/ICAO


Deliverable: CEN-EN-IS. Plan: TC 1996-12, FV 1997-12.

6.5.2 JTC1/SC17 (and CEN TC224)

Description: Standards for IC-cards will be amended to support UCS.


Related: prEN 1923 and ENV 1973.


Deliverables EN-IS. Plan: start 1996, FV 1998.

6.5.3 JTC1/SC18 (and CEN TC304)

Description: This base standard for keyboards should be transposed to avoid national deviations and to meet needs expressed by the CEN committee on ergonomics.


Related: C/2111a, ETSI HF, CENELEC EN 60948, CEN TC122, CEN TC224.


Deliverable: Multi-part EN. Plan: PE 1996-02 FV 1996-09

6.5.4 JTC1/SC21 (and CEN TC304)

C/3331 Introduction of locales in ISO/IEC 9594 The Directory. Phase: 3

Background: Directory services must provide user friendly services in multi-national, multi-lingual and multi-cultural environments. ISO/IEC JTC1 SC21 is working on the enhancement of the Directory to provide multi-locale support. The current work is focused on Attribute with Value Context. One of these Attribute is the Language Attribute which specify the language in which Attribute Types are stored in the Directory Data Base. The other Attribute is Temporal Attribute Value Context which specify the current time (default value). UCS-4 and UCS-2 were introduced also recently. Still full support of all Locale items specified in the European Registry of cultural elements is missing in the Directory. Other important outstanding issue is the application of contexts to Distinguished Names. Further work is required for resolving the issues related to the introduction of AVCs and additional extensions to support the internationalization (matching rules, ordering rules, introduction of flags for critical extensions etc.).


Description: Develop an amendment to ISO/IEC 9594 to include full support for all Locale items specified in the European/International registry of cultural elements. Work to be performed by ISO/IEC JTC1 SC21, input from CEN. Pilot projects are recommended.


Related: C/332.


Deliverable: AMD. to ISO/IEC 9594 (not EN). Plan: CD 1996-01, FV 1997-06.

6.5.5 JTC1/SC22 (and CEN TC304)

Background: Providing input and output in natural languages is a major task in the localization process. In many cases the maker of the application is not able to provide a localized product for all languages of the world, and a standard for specifying messages available to national localizers or users would make localization available in more cultures. Tools for generating and maintaining message catalogues and APIs to interface the message catalogues are also needed . This work is just popping up in WG14 -- C of the API side. This is a somewhat bigger issue, but it may be part of the WG20 LIS API work and then data specification should be done. The service was addressed some years ago in IEEE POSIX, but fell because of disagreement.


Description: Develop an international standard, based on the ENV coming from the Glossasoft work.


Related: ENV.


Deliverable: EN-IS (to replace ENV). Plan: start 1996, FV 1997

L/111382 Deterministic ordering of all UCS characters. Phase: 4.

Description: Define the sorting order of all UCS characters. This work is already underway in WG20, but closer liaison with the CEN work on sorting of the European subsets is needed.


Related: CEN work (L/111381).


Deliverable: EN-IS (replace ENV). Plan: CD/TC 1995-10, FV 1997/8

L/1312 Update POSIX to cover more cultural conventions. Phase: 4.

Description: The formal specification techniques for Locales in POSIX are to be extended to cover more classes of cultural conventions than is possible in the current version. This extension of the POSIX model is very slowly underway in WG20 and WG15 Extend formal specification techniques to cover more classes of cultural conventions


Related: L/31, C/332.


Deliverable: Revized EN-ISO/IEC 9945. Plan: start 1996, FV 1998.

L/132 Formal specification techniques for cultural data (in addition to POSIX). Phase: 4

Description: Develop additional formal specification techniques for cultural data. To be done in co-operation with SC21. Recently approved WI by SC22, CD expected in 1996.


Related: SC21, ISO/IEC 9945-update.


Deliverable: EN-IS. Plan: start 1995, FV 1998.

L/2111b Guidelines on national specifications of cultural conventions. Phase: 3

Background: A NP from WG15 is out for ballot in JTC1, it has passed SC22 ballot recently. The work has been underway for two years, but have not had a lot of progress. European input would be beneficial to the project.


Description: Develop European guidelines that can serve as input to the international work item.


Related: L/2111a.


Deliverable: CR-IS. Plan: TE 1996-06, FV 1996-12.

L/212 International cultural registry. Phase: 3

Description: Other ISO work is dependent on the existence of an international registry rather than a European one, making this is a high priority item. The work on this has been stalled in SC22 and WG20, so the best way to proceed may be to use Vienna agreement or fast-track procedure on the European cultural register.


Related: SC22 work, L/211a (replace ENV)


Deliverable: EN-IS. Plan: PE 1996-03, FV 1997-03.

L/31 Update POSIX to include locale default rules. Phase: 4

Background: POSIX locales are not always available for any language and culture, and a user then has a need for specifying another locale which is acceptable, for example which other language that messages are acceptable in, if the application does not have messages in the native language. This should be multilevel fallback rules, so that the user can specify a set of preferences.


Description: The internationalization model and APIs need to be modified to accommodate this need. New, but small item for WG15.


Related: L/1312, C/332.


Deliverable: Revized EN-ISO/IEC 9945. Plan: start 1996, FV 1998

C/3131 Guide on conversion between UCS coding forms. Phase: 3

Background: A utility "iconv" is defined in POSIX .2b, and an associated API should be defined, of example as part of WG20 internationalization/UCS APIs.


Description: A guide will be developed for implementers of conversion functions in operating systems (such as POSIX iconv). This guide could be input to the WG20 work on APIs for internationalization.


Related: 9995-2b, WG20-APIs.


Deliverable: CR-IS. Plan: TE 1996-06, FV 1996-12.

C/331 Guidelines for UCS in programming languages. Phase: 3.

Description: This guideline should be included in the 2nd revision of TR 10176. PDTR expected autumn 1995.


Related: C/3311a+b.


Deliverable: CR TR 10176 (3rd edition). Plan: CD/TC 1996-10, FV 1997-06.

C/3311a Support for UCS in programming languages. Phase: 4.

Description: Work is underway in many PLs -- some already can handle it, like C, C++, Ada, Prolog.


Related: SC22 groups.


Deliverable: ISs (not to be transposed). Plan: start 1995, FV various.

C/3311b Guidelines for the design of internationalization. Phase: 3.

Description: This guideline is included in the 1st revision of TR 10176, going to CD stage in Oct 1995.


Related: C/331, C/3311a.


Deliverable: CD TR 10176 (2nd edition). Plan: CD/TR 1996-01, FV 1996-07.

C/3312 Language independent API specification for internationalization and UCS. Phase: 4.

Description: Develop an Application Program Interface for support of internationalization and Universal Character Set, independent of programming language. A NWI proposal (subdivision) has been sent by WG20 to SC22.


Related: C/331, C/3311a-b.


Deliverable: EN-IS. Plan: start 1995, FV 1997/8.

C/332 Support for Locale registry in POSIX operating systems. Phase: 3.

Description: This work item will add support for linking to the International registry of Locales in ISO/IEC 9945. Support for the locales themselves is already in C, C++ and POSIX, but work to include support for the registry is not begun.


Related: SC22/WG15.


Deliverable: EN-ISO/IEC 9945 rev. 1. Plan: TC 1997, FV 1998.

6.5.6 JTC1/SC24

7. Description of new EWOS work

C/3332g UCS in MHS. Phase: 3

Background: The inter-operability of MHS applications is considering ISO 646 only. Introduction of UCS, the concept of Locales and internationalization is required for the MHS standards.


Description: Update MHS standards to include support for UCS, Locales and internationalization. EG-MHS to liaise with ETSI and JTC1. CEN can provide input if needed.


Related: ETSI/TE3, JTC1/SC18, CEN/TC304.


Deliverables: ISO standards (not to be transposed). Plan: CD 1996, FV 1998.

C/3332h UCS in ODA. Phase: 3

Background: The latest EWOS document on ODA in the clause on character sets is introducing only 8-bit character sets (ISO 8859) and the code extension techniques. Introduction of UCS, the concept of Locales and internationalization is required for the ODA standards.


Description: Update ODA standards to include support for UCS, Locales and internationalization. EG-SMMI to liaise with ETSI and JTC1. CEN can provide input if needed.


Related: ETSI/TE, JTC1/SC18, CEN/TC304.


Deliverables: ISO standards (not to be transposed). Plan: CD 1996, FV 1998.

C/3332i UCS in SGML. Phase: 3

Description: SGML (and its WWW-implementation HTML) uses its own coding mechanism, which is not compatible with JTC1/SC2 standards. To allow full inter-operability, the SGML coding mechanism should be enhanced to include support for UCS. Work for EWOS-EG SMMI.


Related: ETSI TE, JTC1/SC18, CEN/TC304


Deliverable: ISO standards (not to be transposed). Plan: CD 1996, FV 1998

8. Description of new ETSI work

C/2 Input/output devices for disabled and elderly people. Phase: 3

Background: Input and output methods are required for selection from the repertoire of ISO/IEC 10646 for elderly and disabled persons. The keyboard is not a realistic option as an input method for such selection by persons with physical disabilities, such as reduced or absence of manual exterity. For such persons a screen-based solution needs development with selection by voice, either speech recognition from a matrix on screen, or voice actuated cursor. More severe disabilities may require interfacing to a variety of control devices such as suck-blow, or eye-movements for cursor control. Persons with visual disabilities require an output interface to screen alternatives, such as Braille printers, audio output devices and similar.


Description: A work programme is needed to study and identify the possible input/output devices which give a suitable match to a range of disabilities and their combinations and define suitable interface protocols for connection and signalling in the IT and telecommunications environment. One of the items is a standard method for the special text telephony used by deaf and hearing and speech impaired persons to be developed and standardized. The national 7-bit coded character set used today will be replaced by (preferably) ISO/IEC 10646. A small set of control functions will be developed to control real time text conversations. The system will be able to connect to common electronic mail systems. Work to be performed by ETSI TC-HF and TC-TE.


Related: Inputs from CEN and co-operation with TIDE and similar programs, in particular ISO TC173, CEN TC293 and the ETSI/EU workshop on disabilities planned in 1996.


Deliverables: Multiple ETS/ETR. Plan: various.

C/3332a Guidelines on providing multilingual functionality in ETSI standards. Phase: 3

Background: ISO 10 646 and the internationalized work had very little effect on the telematic and OSI standards. Work was done only within X.500 and VT standards. All other telematics standards (Teletext, Videotext, X.400) or new services coming from the Internet world like World Wide Web, Library Applications, ODA has done almost nothing about that. The latest EWOS document on ODA in the clause on character sets is introducing only 8-bit character sets (ISO 8859) and the code extension techniques. The inter-operability of MHS applications is considering ISO 646 only. Introduction of UCS, the concept of Locales and internationalization is required for all these standards.


Description: A study will analyse Telematic services with respect to how UCS can be implemented in the future standards for these services. It is proposed that the study be carried out by ETSI/HF and the results be made available to all ETSI groups in the form of guidelines for providing multilingual functionality in applications. THIS IS THE MOST URGENT ITEM FOR ETSI. Work for TC-HF -- Project Team including external experts.

Related: All C/3332.

Deliverable: ETR. Plan: TC 1996-01, FV 1996-06.

C/3332c UCS in Digital Video Broadcasting. Phase: 3

Description: Standards for character handling in the future European Digital Video Broadcasting (DVB) systems will be developed. This system will have input from computer systems, and also via receivers forward output to computers. The system will serve many different languages, and therefore must have a repertoire much larger than a single 8-bit character set can offer. -- Work for ETSI/EBU JTC.


Related: EBU, CLC TC106, ENV 1973.


Deliverable: rev. ETS 300468. Plan: TC 1996, FV 1997.

C/3332d UCS in Videotex. Phase: 3

Description: Develop support for UCS in Videotex standards. Work for STC-TE1 in co-operation with ITU.


Related: ENV 1973.


Deliverable: revized ETSs. Plan: TC 1996, FV 1998

C/3332e UCS in radio paging. Phase: 3

Description: Develop support for UCS in radio paging standards. Work for STC-RES4.


Related: GSM, ENV 1973.


Deliverable: ETS (one or more). Plan: TC 1996, FV 1997

C/3332f UCS in text communication over GSM. Phase 3.

Description: Develop support for UCS text communication in GSM standards. Work for TC-SGM.


Related: RES4, ENV 1973.


Deliverable: ETS (one or more). Plan: TC 1996, FV 1997

C/3332g UCS in MHS see clause 7.

C/3332h UCS in ODA see clause 7.

C/3332i UCS in SGML see clause 7.

9. Description of new CENELEC work

C/3332c UCS in Digital Video Broadcasting see clause 8.

C/3332j UCS in RDS. Phase: 4

Description: EN 50067 will be updated to include support for the Universal Character Set, implementing at least the European Subsets of UCS. Work for TC107.


Related: prEN 1923 ENV 1973.


Deliverable: EN. Plan: start 1996/7, FV 1998

C/3332k UCS in HBES. Phase: 3

Description: The higher layers of the HBES architecture (EN 50090) will be updated to include support for at least the European subsets of UCS. Work for TC105.


Related: EN 60948, CEN/TC304


Deliverable: revised EN 50090-1, -2 and -3. Plan: PE 1996-12. FV 1997-12

10. Funding of the work

Annex A

Members of this PT

Annex B

Bibliography

Various published articles on Internationalization of IT.

Additional references are in annexes C-F

Annex C

Mandate M/037

STANDARDIZATION MANDATE ADDRESSED TO CEN/CENELEC/ETSI IN THE AREA OF CHARACTER TECHNOLOGY

PURPOSE

The standardization organizations CEN, CENELEC and ETSI should define a coherent work programme for character repertoires an coding, including a Taxonomy for Character Set Technology.

HISTORY

The last years have seen a major technological evolution, accompanied by an equally important cost reduction, which all together have made information processing very wide spread, and no more the reserved domain of highly qualified specialists.

In parallel, the evolution of the communication concepts, materialized by the OSI standards, has created a new environment which does not constrain a processing system by limiting its communication capabilities to a closed set of proprietary protocols.

Therefore, it is obvious that the definition and coding of the character repertoires have to be revized and a policy must be defined in accordance with the new context and the new requirements.

It is also obvious that the current systems must be adapted to the new situation.

Consequently, character repertoires solutions are required for the new environment, together with a clear migration strategy for existing equipment.

The European standardization organizations have up to now published several standardization documents concerning sets and coding e.g.

a) European standards:

ENV 41501: Information Systems Interconnection -- Character Repertoire and coding for information received form or transmitted to CEPT Videotex services or private videotex systems (REV)

ENV 41502: Information Systems Interconnection -- Character Repertoire and coding for information received from or transmitted to the Teletex Service or private information processing systems using Teletex technology (REV)

ENV 41503: Repertoire and coding of characters -- European Character Repertoire and their coding (REV)

ENV 41504: Information Systems Interconnection -- Character Repertoire and coding inter-working with Telex Services (REV)

ENV 41505: Repertoire and coding of characters -- Graphic Character Repertoire and coding for line drawing (REV)

ENV 41506: Information Systems Interconnection -- Data stream formats for information received from or transmitted to the Teletex Service or private information processing systems using Teletex technology (REV)

ENV 41507: Information Systems Interconnection -- Data stream formats for information received from or transmitted to CEPT Videotex services or private videotex systems (REV)

ENV 41508: Repertoire and coding of characters -- East European graphic character repertoires and their coding (REV)

CEN/CENELEC report R-IT-04. Report on functional standards for character sets and their coding (12/1990)

EWOS report : usage of coded character sets and repertories in EWOS (resulting from BC-IT-SI 009).

• ISO has recently published IS 10 646
  
• A new CEN Technical Committee TC 304 has taken over the activities of the former CEN/CENELEC WG CSC.
  
• EWOS is implementing the results of their report in the context of the EN/ISP production
  
• The EPHOS project is identifying specific needs with respect to Character Repertoires.

a) Define an European policy and strategy with respect to Character Repertoires and their encodings.

b) To align the existing and ongoing standardization activities with this policy.

ORDER

The Commission invites CEN in co-operation with CENELEC, ETSI and EWOS to define an European work programme for Character Repertoires and coding including a taxonomy for character set technology.

The work programme should:

a) be based on a clear definition of European user requirement, such as the user purchasers or production application standards developed by EWOS and ETSI or by applications of guidance such as EPHOS.


When establishing user requirements due note should be taken of existing material which may have been developed within national activities such as Statskontoret Teknisk Norm Nr. 34.1.

b) reflect the commonalty of user requirements and define all required standardization activities in order to realize the user functionality as defined in (a). It will make a clear separation of standardization activities which reflect user requirements and those required for realization of user requirements.

The work programme should also include:


• A strategy to realign the already existing standardization documents;
  
• A taxonomy of Character Set Standardization;
  
• A definition of European Sub-repertoires of ISO 10646 on the basis of the European user requirements as defined in (a).

• CEN, in co-operation with CENELEC, ETSI and EWOS shall present to the Commission a joint and mutually agreed work programme on standardization activities, providing a coherent long term approach and meeting the criteria set in the "ORDER", within 6 months after the acceptance of the present mandate.
  The work programme will include target dates for the different tasks and a work assignment between the interested parties.
  
• CEN, CENELEC, ETSI and EWOS will execute tasks foreseen in the work programme within a time scale to be discussed with the Commission.
  
• The European standards (EN) shall be adopted by the data agreed. At this date, the three linguistic versions (DE, EN, FR) shall be available as well as the correct titles in the other Community languages.
  
• The European standards adopted shall be transposed into national standards and differing national standards shall be withdrawn from the catalogues of the national standards organization in the Member States within six months of their adoption.
  
• Standstill shall apply from the date of receipt of the programme unless the work items listed in the programme have already been accepted by CEN, in which case standstill is already in force.

Annex D

References to standards

D.1 International coding standards

• ISO 646, 2ed, 1983 : ISO 7-Bit Coded Character Set
  
• ISO 2022, 3ed, 1986 : Code Extension Techniques
  
• ISO 4873, 2ed, 1986 : ISO 8-Bit Code Structure and Rules
  
• ISO 6429, 1ed, 1988 : Control Functions for Coded Character Sets
  
• ISO 6936, 2ed, 1988 : Conversion between ISO 646 and CCITT ITA 2
  
• ISO 6937-1, 1ed, 1983 : Coded Character Sets for Text Communication
  
• ISO 6937-2, 1ed, 1983 : CCS for Text Communication -- Latin Characters
  
• ISO 8859-x, 1ed, 1987 : 8-Bit Single-Byte Coded Graphic Character Sets
  
  • 1 Latin Alphabet No. 1
    
  • 2 Latin Alphabet No. 2
    
  • 3 Latin Alphabet No. 3
    
  • 4 Latin Alphabet No. 4
    
  • 5 Latin/Cyrillic Alphabet
    
  • 6 Latin/Arabic Alphabet
    
  • 7 Latin Greek Alphabet
    
  • 8 Latin/Hebrew Alphabet
    
  • 9 Latin Alphabet No. 5
    
  • 10 Latin Alphabet No. 6
• ISO/IEC 10646-1:1993 Universal Character Set -- Part 1: Basic Multilingual Plane
  • Unicode : Unicode is a fixed-width 16-bit multi-lingual character code, which is a part of ISO/IEC 10646-1
• ISO/IEC DIS 10367 : Coded Graphic Character Sets for Use in 8-Bit Codes
  
• ISO 8824, 1ed, 1987 : Abstract Syntax Notation One (ASN.1)
  
• ISO 8825, 1ed, 1987 : Specification of Basic Encoding Rules for ASN.1
  
• ISO 7350, 2nd, 1990 : Registration of Graphic Character Sub-repertoires
  
• ISO 2735, 2nd, 1990 : Registration of the escape sequences with coded character sets
  
• CCITT T.61, 1985 : Character Repertoire and Coded Character Sets for the
  Teletext, Videotex

• ISO 2375, 3ed, 1985 : Procedure for Registration of Escape Sequences
  
• ISO 7350, 2nd, 1990 : Registration of Graphic Character Sub-repertoires

D.2 Regional Standards

• ENV 41 501 : Graphic character repertoire for Videotex systems (replaced by ETS)
  
• ENV 41 502 : Graphic character repertoire for Teletex (replaced by ETS)
  
• ENV 41 503 : European graphic character repertoires and their coding
  
• ENV 41 504 : Character repertoire and coding for inter-working with Telex
  
• ENV 41 505 : Character repertoire and coding for line drawing
  
• ENV 41 506 : Data stream formats compatible with CEPT Teletex (replaced by ETS)
  
• ENV 41 507 : Data stream formats compatible with the CEPT Videotex (repl. by ETS)
  
• ENV 41 508 : East European graphic character repertoires
  
• ENV 41 509 : Formatted documents -- Basic character content
  
• ENV 41 510 : Formatted documents -- Extended mixed mode
  
• ENV 41 511 : Layout independent documents -- Simple messaging profile
  
• EN 50067 : Specification of the Radio Data System (RDS)
  
• EN 60948 : Numeric keyboard for home electronic systems
  
• ETS 300133-2 : European Radio Message System (ERMES)
  
• ETS 300 072 : Videotex presentation layer data syntax
  
• ETS 300 105 : International Videotex inter-working
  
• ETS 300 015 : Basic and recommended additional requirements for terminal equipment supporting Teletex application

• GB 2312:1980 : Basic Chinese character set
  
• GB 12345:1990 : Complex Chinese characters
  
• GB 8565:1989 : Chinese coded character set for text communication
  
• GB 7589:1987 : 7327 additional Chinese characters
  
• GB 7590:1987 : 7039 additional Chinese characters
  
• GB 13000:1990 : Proposal for a unified Han Character Set (HCS)
  
• CNS 11643 : 13.051 most commonly used Chinese characters, which can be presented in BIG-5, 5550, and TCA.

• CNS 11643 : contains 13.051 most commonly used Chinese characters, which
  can be presented in different coding schemes. The most popular internal codes are BIG-5, 5550, and TCA.

• JIS X0208:1990 : Code for the Japanese graphic character set
  
• JIS X0212:1990 : Code of the supplement Japanese graphic character set
  
• JIS X0201:1976 : Code for information interchange in Japan
  
• JIS X0211:1991 : Control functions for coded character sets in Japan
  
• JIS X0202: see ISO 2022
  
• JIS X0301:1979 : Identification Code of Dates
  
• JIS X0302:1977 : Identification Code of Times
  
• JIS X6002:.... : Keyboard Layout for Information Processing Using the JIS 7 Bit Coded Character Set

• KSC 5601:1987 : Korean national character set standard

D.3 Standards related to internationalization

• ISO 1092
  
• ISO 2126
  
• ISO 8884 : Keyboard Layout for Multiple Latin-alphabet Languages (replaced by 9995)
  
• ISO 9241 : Ergonomic aspects of hardware and software products.
  
• ISO/IEC 9995 : Keyboard layouts for text and office systems (multi-part)
  
• CSA-Z243.200 : Canadian French-English keyboard standard
  
• JIS X6002:19xx : Keyboard Layout for JIS 7-bit coded character sets

• ISO 9945-1, 1ed, 1990 : POSIX -- System Application Program Interface
  
• IEEE P1003.2 : POSIX shell and utilities

• ISO 8601, 1ed, 1988 : Representation of Dates and Times (replaces ISO 2014)
  
• JIS X0301:1977 : Identification Code of Dates in Japan

• ISO 3307, 1ed, 1975 : Representation of Time of the Day
  
• ISO 4031, 1ed, 1987 : Representation of Local Time Differentials
  
• ISO 8601, 1ed, 1988 : Representation of Dates and Times
  
• JIS X0302:1977 : Identification Code of Times in Japan

• ISO 6093, 1ed, 1985 : Presentation of Numerical Values

• ISO 4217, 3ed, 1987 : Codes for the Representation of Currencies
  
• ISO 6093, 1ed, 1985 : Presentation of Numerical Values in Character Funds

• National standards

D.4 Standards related to the visual representation of characters (glyphs) :

• ISO/IEC 1073-1 Alphanumeric Character Sets for Optical Recognition
  
  • Part 1: Character set OCR-A.
    
  • Part 2: Character set OCR-B (under revision)
  
  
• ISO/IEC 9541-1 Font information interchange
  
  • Part 1: Architecture
    
  • Part 2: Interchange format
    
  • Part 3: Glyph shape representation
  
  
• ISO/IEC 10036 Font information interchange -- Procedure for registration of glyph and glyph collection identifiers.

D.5 Other standards:

• 1.2 Introduction and General Principles
  
• 2.2 Document Structures
  
• 3 Document Processing Reference Model
  
• 4.2 Document Profile
  
• 5.2 Office Document Interchange Format
  
• 6.2 Character Content Architecture
  
• 7 Raster Graphics Content Architecture
  
• 8 Geometric Graphics Content Architecture

Annex E

General recommendations of the CST-workshop

GENERAL SITUATION:

EUROPEAN WORK TO BE DONE:

Annex F

The indigenous languages of Europe

The information in F.9 is taken from ENV 1973 and lists only those languages covered by the Minimum European Subset of UCS.

F.1 Afro-Asiatic languages

Coptic, Maltese

F.2 Basque

Basque

F.3 Caucasian languages

North Caucasian

Andi, Axvax, Bagulal, Botlix, Chamalal, Godoberi, Karata, Tindi, Avar, Bezhta, Hunzib, Tsez, Hinux, Xvarshi, Dargwa, Kubax, Lak, Archi, Agul, Budux, Kryts, Lezgi, Rutul, Tabasaran, Tsaxur, Udi, Xinalug, Bats, Chechen, Ingush, Abaza, Abxaz, Adygh, Kabardian, Ubyx
South Caucasian

Georgian, Svan, Laz, Mingrelian

F.4 Eskimo-Aleut languages

Greenlandic

F.5 Indo-European languages

Albanian

Albanian, Arvanite
Armenian

Armenian
Baltic

Latvian, Lithuanian
Celtic

Breton, Cornish, Welsh, Irish Gaelic, Manx Gaelic, Scottish Gaelic
Germanic

Danish, Älvdalska, Swedish, Faroese, Icelandic, Norwegian, German, Lëtzebuergesh, Swiss German, Yiddish, Dutch/Flemish, Low German, English, Lallans, East Frisian, North Frisian, West Frisian
Greek

Greek, Tsakonian
Indo-Iranian

Romany, Ossetian, Kurdish, Talysh, Tat
Italic

Latin, Esperanto, Istro-Romanian, Romanian, Arumanian, Megleno-Romanian, Franco-Provençal, French, Walloon, Occitan, Aragonese, Ladino, Spanish, Catalan, Asturian, Galician, Portuguese, Corsican, Italian, Friulian, Ladin, Romansch, Sardinian
Slavic

Belorussian, Russian, Ukrainian, Rusinian, Bulgarian, Macedonian, Pomak, Old Church Slavonic, Croatian, Serbian, Slovene, Lower Sorbian, Upper Sorbian, Kashubian, Polish, Czech, Slovak

F.6 Mongolian languages

Kalmyk

F.7 Turkic languages

Chuvash, Kazakh, Nogai, Azerbaijani, Crimean Turkish, Gagauz, Turkish, Turkmen, Bashkir, Karachay-Balkar, Karaim, Kumyk, Baraba, Crimean Tatar, Kazan Tatar

F.8 Uralic languages

Finno-Ugric

Finnish, Ingrian, Karelian, Ludic, Olonets, Veps, Estonian, Livonian, Votic, Lule Sámi, North Sámi, Pite Sámi, Inari Sámi, Kildin Sámi, Skolt Sámi, Ter Sámi, South Sámi, Ume Sámi, East Mari, West Mari, Erza, Moksha, Komi, Komi-Permyak, Udmurt, Hungarian
Samoyed

Nenets

F.9 Languages covered by the Minimum European Subset (MES)

Latin script:
Albanian, Basque, Breton, Catalan, Croatian, Czech, Danish, Dutch, English, Estonian, Faroese, Finnish, French, Frisian, Friulian, Gaelic, Galician, German, Greenlandic, Hungarian, Icelandic, Irish Gaelic, Italian, Ladin, Latin, Latvian, Lëtzebuergesh, Lithuanian, Livonian, Maltese, Manx Gaelic, Norwegian, Polish, Portuguese, Romanian, Rumantsch, Sámi languages, Spanish, Swedish, Turkish, Welsh.
Cyrillic script:
Bulgarian, Belorussian, Macedonian, Russian, Serbian, Ukrainian.
Greek script:
Greek.

Annex G

Extract from the Nordic report on cultural requirements [5]

G.0 Table of contents

• 1.1 Scope and field of application
  
• 1.2 Definitions
  
• 1.3 IT-terminology
  
• 1.4 Countries, cultures and languages
  
  • 1.4.1 General
    
  • 1.4.2 Countries and their codes
    
  • 1.4.3 Languages and their codes
    
  • 1.4.4 Geographic area of countries and languages
    
  • 1.4.5 Non-Nordic ethnic groups
    
  • 1.4.6 Groupings of the Nordic languages
    
• 1.5 Persons with disabilities
  
  • 1.5.1 A relative concept
    
  • 1.5.2 Computer accessibility
    
    

• 2.1 General
  
• 2.2 Definitions
  
• 2.3 National letters and character sets in use
  
• 2.4 Sorting and searching rules
  
  • 2.4.0 Introduction
    
  • 2.4.1 Sorting by levels
    
  • 2.4.2 Normalization for sorting
    
  • 2.4.3 Searching
    
• 2.5 Transformation of characters
  
  • 2.5.1 Different scripts
    
  • 2.5.2 Fallback notation for Nordic names
    
• 2.6 Properties of characters
  
• 2.7 Use of special characters
  
  • 2.7.1 Hyphens and dashes
    
  • 2.7.2 Quotations
    
  • 2.7.3 Various punctuation marks
    
• 2.8 Character rendition
  
  

• 3.1 General
  
• 3.2 Definitions
  
• 3.3 Personal names
  
  • 3.3.1 The general rule
    
  • 3.3.2 Icelandic names
    
• 3.4 Inflection
  
• 3.5 Hyphenation
  
• 3.6 Spelling
  
  

• 4.1 General
  
• 4.2 Definitions
  
• 4.3 Numbers, ordinals and measures
  
• 4.4 Monetary amounts
  
• 4.5 Date and time
  
  • 4.5.1 Date
    
  • 4.5.2 Time of day
    
  • 4.5.3 Periods of time
    
  • 4.5.4 Numbering of weeks
    
  • 4.5.5 Names of weekdays and months
    
• 4.6 Coding of national entities
  
  • 4.6.1 Country and language
    
  • 4.6.2 Currency
    
  • 4.6.3 Locations
    
  • 4.6.4 Time-zones
    
• 4.7 Telephone numbers
  
• 4.8 Mail-addresses
  
• 4.9 Identification of persons and organizations (was 3.4)
  
• 4.10 Electronic mail addresses
  
• 4.11 Payment account numbers
  
  

• 5.1 General
  
• 5.2 Definitions
  
• 5.3 National and international keyboard layouts
  
• 5.4 Paper
  
• 5.5 The man/machine dialogue
  
• 5.6 Terminal equipment
  
  

• 6.1 General
  
• 6.2 Definitions
  
• 6.3 Library applications
  
• 6.4 Office systems
  
• 6.5 Accounting
  
• 6.6 Wireless applications
  
• 6.7 Government and administration
  
  

• 7.1 General
  
• 7.2 Definitions
  
• 7.3 Procurement
  
• 7.4 Ergonomic and other health requirements
  
• 7.5 Personal privacy requirements