]> Amazon Web Services (AWS)

yakovsh@amazon.com

This RFC documents the common format used for Comma-Separated Values (CSV) files and updates the associated MIME type "text/csv".

Introduction The comma separated values format (CSV) has been used as a common way to exchange data between disparate systems and applications for many years. Surprisingly, while this format is very popular, it has never been formally documented and didn't have a media type registered. This was addressed in 2005 via publication of and the concurrent registration of the "text/csv" media type. Since the publication of , the CSV format has evolved and this specification seeks to reflect these changes as well as update the "text/csv" media type registration.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Motivation For and Status of This Document The original motivation of was to provide a reference in order to register the media type "text/csv". It tried to document existing practices at the time based on the approaches used by most implementations. This document continues to do the same, and updates the original document to reflect current practices for generating and consuming of CSV files. Both and this document are published as informational RFC for the benefit of the Internet community and and not intended to be used as formal standards. Implementers should consult and for crucial differences between IETF standards and informational RFCs.

Definition of the CSV Format While there had been various specifications and implementations for the CSV format (for ex. , , and )), prior to publication of there is no attempt to provide a common specification. This section documents the format that seems to be followed by most implementations (incorporating changes since the publication of ).

High level description

1. Each record is located on a separate line, ended by a line break (CR, LF or CRLF). For example: aaa,bbb,cccCRLF
   zzz,yyy,xxxCRLF
2. The last record in the file MUST have an ending line break. For example: aaa,bbb,cccCRLF
   zzz,yyy,xxxCRLF
3. The first record in the file MAY be an optional header with the same format as normal records. This header will contain names corresponding to the fields in the file and SHOULD contain the same number of fields as the records in the rest of the file. For example: field_name_1,field_name_2,field_name_3CRLF
   aaa,bbb,cccCRLF
   zzz,yyy,xxxCRLF
4. Within each record, there MAY be one or more fields, separated by commas. Each record SHOULD contain the same number of fields throughout the file. Spaces are considered part of a field and SHOULD NOT be ignored. The last field in the record MUST NOT be followed by a comma. For example: aaa,bbb,cccCRLF
5. Each field MAY be enclosed in double quotes (however some programs, do not use double quotes at all). If fields are not enclosed with double quotes, then double quotes MUST NOT appear inside the fields. For example: "aaa","bbb","ccc"CRLF
   zzz,yyy,xxxCRLF
6. Fields containing line breaks (CR, LF or CRLF), double quotes, or commas MUST be enclosed in double-quotes. The same applies for the first field of a record that starts with a hash to avoid the field from being parsed as a comment. For example: "aaa","b CRLF
   bb","ccc"CRLF
   zzz,yyy,xxxCRLF
   "#aaa",#bbb,cccCRLF
7. A double-quote appearing inside a field MUST be escaped by preceding it with another double quote. For example: "aaa","b""bb","ccc"CRLF
8. A hash sign MAY be used to mark lines that are meant to be commented lines. A commented line can contain any whitespace or visible character until it is terminated by a line break (CR, LF or CRLF). A comment line MAY appear in any line of the file (before or after an OPTIONAL header) but MUST NOT be mistaken with a subsequent line of a multi-line field. Subsequent lines of multi-line fields can start with a hash sign and MUST NOT interpreted as comments. For example: #commentCRLF
   aaa,bbb,cccCRLF
   #comment 2CRLF
   "aaa","this is CRLF
   # not a comment","ccc"CRLF

Default charset and line break values Since the initial publication of , the default charset for "text/*" media types has been changed to UTF-8 (as per ) and . This document reflects this change and the default charset for CSV files is now UTF-8. Although section 4.1.1. of defines CRLF to denote line breaks, implementers MAY recognize a single CR or LF as a line break (similar to section 3.1.1.3 of ). However, some implementations MAY use other values.

ABNF Grammar The ABNF grammar (as per ) appears as follows: Note that the authoritative definition of UTF-8 is in section 2.5 of .

Common implementation concerns This section describes some common concerns that may arise when producing or parsing CSV files. All of these remain out of scope for this document and are included for awareness. Implementers may also use other means to handle these use cases such as .

Null values Some implementations (such as databases) treat empty fields and null values differently. For these implementations, there is a need to define a special value representing a null. However, this specification does not attempt to define a default value for nulls. Example of a CSV file with nulls (if "NULL" is used to mark nulls): field_name_1,field_name_2,field_name_3CRLF
aaa,bbb,cccCRLF
zzz,NULL,xxxCRLF

Empty files Implementers should be aware that in accordance to this specification a file does not need to contain any comments or records. Therefore, an empty file with zero bytes is considered valid.

Empty lines This specification recommends but doesn't require having the same number of fields in every line. This allows CSV files to have empty lines without any fields at all. Implementors may choose to to skip empty lines instead of parsing them but this specification does not dictate such behavior. Example of a CSV file with empty lines: field_name_1,field_name_2,field_name_3CRLF
aaa,bbb,cccCRLF
CRLF
zzz,yyy,xxxCRLF However, if the records are only made up of one field it is not possible to differentiate between an empty line, and an empty and unquoted field. This differentiation might play an important role in some implementations such as database exports/imports. Example of a CSV file with empty lines and only one field per record: aaa
CRLF
bbbCRLF

Fields spanning multiple lines When quoted fields are used, it is possible for a field to span multiple lines, even when line breaks appear within such field.

Unique header names Implementers should be aware that some applications may treat header values as unique (either case-sensitive or case-insensitive).

Whitespace outside of quoted fields When quoted fields are used, this document does not allow whitespace between double quotes and commas. Implementers should be aware that some applications may be more lenient and allow whitespace outside the double quotes.

Other field separators This document defines a comma as a field separator but implementers should be aware that some applications may use different values, especially with non-English languages. Those are outside the scope of this document and implementers should consult other efforts such as .

Escaping double quotes This document prescribes that a double-quote appearing inside a field must be escaped by preceding it with another double quote. Implementers should be aware that some applications may choose to use a different escaping mechanism.

BOM header Applications that create text files with unicode character encoding might write a BOM (byte order mark) header in order to support multiple unicode encodings (like UTF-16 and UTF-32). Some applications might be able to read and properly interpret such a header, others could break. Implementors should review section 6 of and section 23.8 of .

Bidirectional text While most of the world's written languages are displayed left-to-right, many languages such as ones based on Hebrew or Arabic scripts are displayed primarily right-to-left. Implementers should consult the "bidirectional display" part in section 5 of for further guidance.

Update to MIME Type Registration of text/csv The media type registration of "text/csv" should be updated as per specific fields below: Encoding considerations:

• CSV MIME entities can consist of binary data as per section 4.8 of . Although section 4.1.1. of defines CRLF to denote line breaks, implementers MAY recognize a single CR or LF as a line break (similar to section 3.1.1.3 of ). However, some implementations may use other values.

Published specification:

• While numerous private specifications exist for various programs and systems, there is no single "master" specification for this format. An attempt at a common definition can be found in and this document. Implementers should note that both documents are informational in nature and are not standards.

Optional parameters: charset

• The "charset" parameter specifies the charset employed by the CSV content. In accordance with , the charset parameter SHOULD be used, and if it is not present, UTF-8 SHOULD be assumed as the default (this implies that US- ASCII CSV will work, even when not specifying the "charset" parameter). Any charset defined by IANA for the "text" tree may be used in conjunction with the "charset" parameter.

Security considerations:

• Text/csv consists of nothing but passive text data that should not pose any direct risks. However, it is possible that malicious data may be included in order to exploit buffer overruns or other bugs in the program processing the text/csv data.

• Implementers and users should also be aware that some software applications may interpret certain characters in the beginning of CSV fields as referring to code or formulas, thus resulting in malicious code execution. This is known as "CSV injection" and users consuming CSV files should filter out such characters.

• The text/csv format provides no confidentiality or integrity protection, so if such protections are needed they must be supplied externally.

• The fact that software implementing fragment identifiers for CSV and software not implementing them differs in behavior, and the fact that different software may show documents or fragments to users in different ways, can lead to misunderstandings on the part of users. Such misunderstandings might be exploited in a way similar to spoofing or phishing.

• Implementers and users of fragment identifiers for CSV text should also be aware of the security considerations in RFC 3986 and RFC 3987 .

Interoperability considerations:

• Due to lack of a single specification, there are considerable differences among implementations. Implementers should "be conservative in what you do, be liberal in what you accept from others" () when processing CSV files. An attempt at a common definition can be found in Section 2.

IANA Considerations IANA is directed to update the MIME type registration for "text/csv" as per instructions provided in of this document and include a reference to this document within the registration.

Security Considerations All security considerations discussed in still apply.

Acknowledgments In addition to everyone thanked previously in , the author would like to thank acknowledge the contributions of the following people to this document: Alperen Belgic, Abed BenBrahim, Damon Koach, Barry Leiba, Oliver Siegmar, Marco Diniz Sousa and Greg Skinner. A special thank you to L.T.S.

References Normative References This RFC documents the format used for Comma-Separated Values (CSV) files and registers the associated MIME type "text/csv". This memo provides information for the Internet community. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document changes RFC 2046 rules regarding default "charset" parameter values for "text/*" media types to better align with common usage by existing clients and servers. [STANDARDS-TRACK] This memo defines URI fragment identifiers for text/csv MIME entities. These fragment identifiers make it possible to refer to parts of a text/csv MIME entity identified by row, column, or cell. Fragment identification can use single items or ranges. This second document defines the general structure of the MIME media typing system and defines an initial set of media types. [STANDARDS-TRACK] The Hypertext Transfer Protocol (HTTP) is a stateless \%application- level protocol for distributed, collaborative, hypertext information systems. This document defines the semantics of HTTP/1.1 messages, as expressed by request methods, request header fields, response status codes, and response header fields, along with the payload of messages (metadata and body content) and mechanisms for content negotiation. Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] This document defines procedures for the specification and registration of media types for use in HTTP, MIME, and other Internet protocols. This memo documents an Internet Best Current Practice. Informative References Creativyst, Inc. W3C Edoceo, Inc. The Unicode Consortium This document discusses the relationship of the Request for Comments (RFCs) notes to Internet Standards. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. This memo documents the process used by the Internet community for the standardization of protocols and procedures. It defines the stages in the standardization process, the requirements for moving a document between stages and the types of documents used during this process. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. This document provides a list of terms used in the IETF when discussing internationalization. The purpose is to help frame discussions of internationalization in the various areas of the IETF and to help introduce the main concepts to IETF participants. This memo documents an Internet Best Current Practice. A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme. [STANDARDS-TRACK] This document defines a new protocol element, the Internationalized Resource Identifier (IRI), as a complement of the Uniform Resource Identifier (URI). An IRI is a sequence of characters from the Universal Character Set (Unicode/ISO 10646). A mapping from IRIs to URIs is defined, which means that IRIs can be used instead of URIs, where appropriate, to identify resources. The approach of defining a new protocol element was chosen instead of extending or changing the definition of URIs. This was done in order to allow a clear distinction and to avoid incompatibilities with existing software. Guidelines are provided for the use and deployment of IRIs in various protocols, formats, and software components that currently deal with URIs.

Major changes since

• Added a section clarifying motivation for this document and standards status
• Changing default encoding to UTF-8 and adding Unicode to the ABNF grammar
• Allowing CR, LF and CRLF for line breaks
• Allowing HTAB in text data
• Mandating a line break at the end of the last line in the file
• Making records and headers optional, thus allowing for an empty file
• Adding definition of commented lines
• Adding a section on common implementation concerns
• Removed "header" parameter for the MIME type since it is not used

Changes since the -00 draft

• Added CSV injection to security considerations (#30
• Added a reference to RFC 7111 (#27)

Changes since the -01 draft

• No changes yet, refreshed to keep draft alive

Changes since the -02 draft

• Refreshed to keep draft alive
• Contact information and Github link changes
• Minor updates on language
• Added a section on bidi handling

Note to Readers

• Note to the RFC Editor: Please remove this section prior to publication.

Development of this draft takes place on Github at: https://github.com/yakovsh/rfc4180-bis Comments can also be sent to the ART mailing list at: https://www.ietf.org/mailman/listinfo/art Full list of changes can be viewed via the IETF document tracker: https://tools.ietf.org/html/draft-shafranovich-rfc4180-bis