13 YAML is a human readable data serialization language. The full YAML language
14 spec can be read at `yaml.org
15 <http://www.yaml.org/spec/1.2/spec.html#Introduction>`_. The simplest form of
16 yaml is just "scalars", "mappings", and "sequences". A scalar is any number
17 or string. The pound/hash symbol (#) begins a comment line. A mapping is
18 a set of key-value pairs where the key ends with a colon. For example:
26 A sequence is a list of items where each item starts with a leading dash ('-').
36 You can combine mappings and sequences by indenting. For example a sequence
37 of mappings in which one of the mapping values is itself a sequence:
41 # a sequence of mappings with one key's value being a sequence
54 Sometime sequences are known to be short and the one entry per line is too
55 verbose, so YAML offers an alternate syntax for sequences called a "Flow
56 Sequence" in which you put comma separated sequence elements into square
57 brackets. The above example could then be simplified to :
62 # a sequence of mappings with one key's value being a flow sequence
68 cpus: [ PowerPC, x86 ]
71 Introduction to YAML I/O
72 ========================
74 The use of indenting makes the YAML easy for a human to read and understand,
75 but having a program read and write YAML involves a lot of tedious details.
76 The YAML I/O library structures and simplifies reading and writing YAML
79 YAML I/O assumes you have some "native" data structures which you want to be
80 able to dump as YAML and recreate from YAML. The first step is to try
81 writing example YAML for your data structures. You may find after looking at
82 possible YAML representations that a direct mapping of your data structures
83 to YAML is not very readable. Often the fields are not in the order that
84 a human would find readable. Or the same information is replicated in multiple
85 locations, making it hard for a human to write such YAML correctly.
87 In relational database theory there is a design step called normalization in
88 which you reorganize fields and tables. The same considerations need to
89 go into the design of your YAML encoding. But, you may not want to change
90 your exisiting native data structures. Therefore, when writing out YAML
91 there may be a normalization step, and when reading YAML there would be a
92 corresponding denormalization step.
94 YAML I/O uses a non-invasive, traits based design. YAML I/O defines some
95 abstract base templates. You specialize those templates on your data types.
96 For instance, if you have an eumerated type FooBar you could specialize
97 ScalarEnumerationTraits on that type and define the enumeration() method:
101 using llvm::yaml::ScalarEnumerationTraits;
102 using llvm::yaml::IO;
105 struct ScalarEnumerationTraits<FooBar> {
106 static void enumeration(IO &io, FooBar &value) {
112 As with all YAML I/O template specializations, the ScalarEnumerationTraits is used for
113 both reading and writing YAML. That is, the mapping between in-memory enum
114 values and the YAML string representation is only in place.
115 This assures that the code for writing and parsing of YAML stays in sync.
117 To specify a YAML mappings, you define a specialization on
118 llvm::yaml::MapppingTraits.
119 If your native data structure happens to be a struct that is already normalized,
120 then the specialization is simple. For example:
124 using llvm::yaml::MapppingTraits;
125 using llvm::yaml::IO;
128 struct MapppingTraits<Person> {
129 static void mapping(IO &io, Person &info) {
130 io.mapRequired("name", info.name);
131 io.mapOptional("hat-size", info.hatSize);
136 A YAML sequence is automatically infered if you data type has begin()/end()
137 iterators and a push_back() method. Therefore any of the STL containers
138 (such as std::vector<>) will automatically translate to YAML sequences.
140 Once you have defined specializations for your data types, you can
141 programmatically use YAML I/O to write a YAML document:
145 using llvm::yaml::Output;
153 std::vector<Person> persons;
154 persons.push_back(tom);
155 persons.push_back(dan);
157 Output yout(llvm::outs());
160 This would write the following:
169 And you can also read such YAML documents with the following code:
173 using llvm::yaml::Input;
175 typedef std::vector<Person> PersonList;
176 std::vector<PersonList> docs;
178 Input yin(document.getBuffer());
184 // Process read document
185 for ( PersonList &pl : docs ) {
186 for ( Person &person : pl ) {
187 cout << "name=" << person.name;
191 One other feature of YAML is the ability to define multiple documents in a
192 single file. That is why reading YAML produces a vector of your document type.
199 When parsing a YAML document, if the input does not match your schema (as
200 expressed in your XxxTraits<> specializations). YAML I/O
201 will print out an error message and your Input object's error() method will
202 return true. For instance the following document:
211 Has a key (shoe-size) that is not defined in the schema. YAML I/O will
212 automatically generate this error:
216 YAML:2:2: error: unknown key 'shoe-size'
220 Similar errors are produced for other input not conforming to the schema.
226 YAML scalars are just strings (i.e. not a sequence or mapping). The YAML I/O
227 library provides support for translating between YAML scalars and specific
233 The following types have built-in support in YAML I/O:
248 That is, you can use those types in fields of MapppingTraits or as element type
249 in sequence. When reading, YAML I/O will validate that the string found
250 is convertible to that type and error out if not.
255 Given that YAML I/O is trait based, the selection of how to convert your data
256 to YAML is based on the type of your data. But in C++ type matching, typedefs
257 do not generate unique type names. That means if you have two typedefs of
258 unsigned int, to YAML I/O both types look exactly like unsigned int. To
259 facilitate make unique type names, YAML I/O provides a macro which is used
260 like a typedef on built-in types, but expands to create a class with conversion
261 operators to and from the base type. For example:
265 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyFooFlags)
266 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyBarFlags)
268 This generates two classes MyFooFlags and MyBarFlags which you can use in your
269 native data structures instead of uint32_t. They are implicitly
270 converted to and from uint32_t. The point of creating these unique types
271 is that you can now specify traits on them to get different YAML conversions.
275 An example use of a unique type is that YAML I/O provides fixed sized unsigned
276 integers that are written with YAML I/O as hexadecimal instead of the decimal
277 format used by the built-in integer types:
284 You can use llvm::yaml::Hex32 instead of uint32_t and the only different will
285 be that when YAML I/O writes out that type it will be formatted in hexadecimal.
288 ScalarEnumerationTraits
289 -----------------------
290 YAML I/O supports translating between in-memory enumerations and a set of string
291 values in YAML documents. This is done by specializing ScalarEnumerationTraits<>
292 on your enumeration type and define a enumeration() method.
293 For instance, suppose you had an enumeration of CPUs and a struct with it as
309 To support reading and writing of this enumeration, you can define a
310 ScalarEnumerationTraits specialization on CPUs, which can then be used
315 using llvm::yaml::ScalarEnumerationTraits;
316 using llvm::yaml::MapppingTraits;
317 using llvm::yaml::IO;
320 struct ScalarEnumerationTraits<CPUs> {
321 static void enumeration(IO &io, CPUs &value) {
322 io.enumCase(value, "x86_64", cpu_x86_64);
323 io.enumCase(value, "x86", cpu_x86);
324 io.enumCase(value, "PowerPC", cpu_PowerPC);
329 struct MapppingTraits<Info> {
330 static void mapping(IO &io, Info &info) {
331 io.mapRequired("cpu", info.cpu);
332 io.mapOptional("flags", info.flags, 0);
336 When reading YAML, if the string found does not match any of the the strings
337 specified by enumCase() methods, an error is automatically generated.
338 When writing YAML, if the value being written does not match any of the values
339 specified by the enumCase() methods, a runtime assertion is triggered.
344 Another common data structure in C++ is a field where each bit has a unique
345 meaning. This is often used in a "flags" field. YAML I/O has support for
346 converting such fields to a flow sequence. For instance suppose you
347 had the following bit flags defined:
358 LLVM_YAML_UNIQUE_TYPE(MyFlags, uint32_t)
360 To support reading and writing of MyFlags, you specialize ScalarBitSetTraits<>
361 on MyFlags and provide the bit values and their names.
365 using llvm::yaml::ScalarBitSetTraits;
366 using llvm::yaml::MapppingTraits;
367 using llvm::yaml::IO;
370 struct ScalarBitSetTraits<MyFlags> {
371 static void bitset(IO &io, MyFlags &value) {
372 io.bitSetCase(value, "hollow", flagHollow);
373 io.bitSetCase(value, "flat", flagFlat);
374 io.bitSetCase(value, "round", flagRound);
375 io.bitSetCase(value, "pointy", flagPointy);
385 struct MapppingTraits<Info> {
386 static void mapping(IO &io, Info& info) {
387 io.mapRequired("name", info.name);
388 io.mapRequired("flags", info.flags);
392 With the above, YAML I/O (when writing) will test mask each value in the
393 bitset trait against the flags field, and each that matches will
394 cause the corresponding string to be added to the flow sequence. The opposite
395 is done when reading and any unknown string values will result in a error. With
396 the above schema, a same valid YAML document is:
401 flags: [ pointy, flat ]
406 Sometimes for readability a scalar needs to be formatted in a custom way. For
407 instance your internal data structure may use a integer for time (seconds since
408 some epoch), but in YAML it would be much nicer to express that integer in
409 some time format (e.g. 4-May-2012 10:30pm). YAML I/O has a way to support
410 custom formatting and parsing of scalar types by specializing ScalarTraits<> on
411 your data type. When writing, YAML I/O will provide the native type and
412 your specialization must create a temporary llvm::StringRef. When reading,
413 YAML I/O will provide a llvm::StringRef of scalar and your specialization
414 must convert that to your native data type. An outline of a custom scalar type
419 using llvm::yaml::ScalarTraits;
420 using llvm::yaml::IO;
423 struct ScalarTraits<MyCustomType> {
424 static void output(const T &value, llvm::raw_ostream &out) {
425 out << value; // do custom formatting here
427 static StringRef input(StringRef scalar, T &value) {
428 // do custom parsing here. Return the empty string on success,
429 // or an error message on failure.
438 To be translated to or from a YAML mapping for your type T you must specialize
439 llvm::yaml::MapppingTraits on T and implement the "void mapping(IO &io, T&)"
440 method. If your native data structures use pointers to a class everywhere,
441 you can specialize on the class pointer. Examples:
445 using llvm::yaml::MapppingTraits;
446 using llvm::yaml::IO;
448 // Example of struct Foo which is used by value
450 struct MapppingTraits<Foo> {
451 static void mapping(IO &io, Foo &foo) {
452 io.mapOptional("size", foo.size);
457 // Example of struct Bar which is natively always a pointer
459 struct MapppingTraits<Bar*> {
460 static void mapping(IO &io, Bar *&bar) {
461 io.mapOptional("size", bar->size);
470 The mapping() method is responsible, if needed, for normalizing and
471 denormalizing. In a simple case where the native data structure requires no
472 normalization, the mapping method just uses mapOptional() or mapRequired() to
473 bind the struct's fields to YAML key names. For example:
477 using llvm::yaml::MapppingTraits;
478 using llvm::yaml::IO;
481 struct MapppingTraits<Person> {
482 static void mapping(IO &io, Person &info) {
483 io.mapRequired("name", info.name);
484 io.mapOptional("hat-size", info.hatSize);
492 When [de]normalization is required, the mapping() method needs a way to access
493 normalized values as fields. To help with this, there is
494 a template MappingNormalization<> which you can then use to automatically
495 do the normalization and denormalization. The template is used to create
496 a local variable in your mapping() method which contains the normalized keys.
498 Suppose you have native data type
499 Polar which specifies a position in polar coordinates (distance, angle):
508 but you've decided the normalized YAML for should be in x,y coordinates. That
509 is, you want the yaml to look like:
516 You can support this by defining a MapppingTraits that normalizes the polar
517 coordinates to x,y coordinates when writing YAML and denormalizes x,y
518 coordindates into polar when reading YAML.
522 using llvm::yaml::MapppingTraits;
523 using llvm::yaml::IO;
526 struct MapppingTraits<Polar> {
528 class NormalizedPolar {
530 NormalizedPolar(IO &io)
533 NormalizedPolar(IO &, Polar &polar)
534 : x(polar.distance * cos(polar.angle)),
535 y(polar.distance * sin(polar.angle)) {
537 Polar denormalize(IO &) {
538 return Polar(sqrt(x*x+y*y, arctan(x,y));
545 static void mapping(IO &io, Polar &polar) {
546 MappingNormalization<NormalizedPolar, Polar> keys(io, polar);
548 io.mapRequired("x", keys->x);
549 io.mapRequired("y", keys->y);
553 When writing YAML, the local variable "keys" will be a stack allocated
554 instance of NormalizedPolar, constructed from the suppled polar object which
555 initializes it x and y fields. The mapRequired() methods then write out the x
556 and y values as key/value pairs.
558 When reading YAML, the local variable "keys" will be a stack allocated instance
559 of NormalizedPolar, constructed by the empty constructor. The mapRequired
560 methods will find the matching key in the YAML document and fill in the x and y
561 fields of the NormalizedPolar object keys. At the end of the mapping() method
562 when the local keys variable goes out of scope, the denormalize() method will
563 automatically be called to convert the read values back to polar coordinates,
564 and then assigned back to the second parameter to mapping().
566 In some cases, the normalized class may be a subclass of the native type and
567 could be returned by the denormalize() method, except that the temporary
568 normalized instance is stack allocated. In these cases, the utility template
569 MappingNormalizationHeap<> can be used instead. It just like
570 MappingNormalization<> except that it heap allocates the normalized object
571 when reading YAML. It never destroyes the normalized object. The denormalize()
572 method can this return "this".
577 Within a mapping() method, calls to io.mapRequired() mean that that key is
578 required to exist when parsing YAML documents, otherwise YAML I/O will issue an
581 On the other hand, keys registered with io.mapOptional() are allowed to not
582 exist in the YAML document being read. So what value is put in the field
583 for those optional keys?
584 There are two steps to how those optional fields are filled in. First, the
585 second parameter to the mapping() method is a reference to a native class. That
586 native class must have a default constructor. Whatever value the default
587 constructor initially sets for an optional field will be that field's value.
588 Second, the mapOptional() method has an optional third parameter. If provided
589 it is the value that mapOptional() should set that field to if the YAML document
590 does not have that key.
592 There is one important difference between those two ways (default constructor
593 and third parameter to mapOptional). When YAML I/O generates a YAML document,
594 if the mapOptional() third parameter is used, if the actual value being written
595 is the same as (using ==) the default value, then that key/value is not written.
601 When writing out a YAML document, the keys are written in the order that the
602 calls to mapRequired()/mapOptional() are made in the mapping() method. This
603 gives you a chance to write the fields in an order that a human reader of
604 the YAML document would find natural. This may be different that the order
605 of the fields in the native class.
607 When reading in a YAML document, the keys in the document can be in any order,
608 but they are processed in the order that the calls to mapRequired()/mapOptional()
609 are made in the mapping() method. That enables some interesting
610 functionality. For instance, if the first field bound is the cpu and the second
611 field bound is flags, and the flags are cpu specific, you can programmatically
612 switch how the flags are converted to and from YAML based on the cpu.
613 This works for both reading and writing. For example:
617 using llvm::yaml::MapppingTraits;
618 using llvm::yaml::IO;
626 struct MapppingTraits<Info> {
627 static void mapping(IO &io, Info &info) {
628 io.mapRequired("cpu", info.cpu);
629 // flags must come after cpu for this to work when reading yaml
630 if ( info.cpu == cpu_x86_64 )
631 io.mapRequired("flags", *(My86_64Flags*)info.flags);
633 io.mapRequired("flags", *(My86Flags*)info.flags);
641 To be translated to or from a YAML sequence for your type T you must specialize
642 llvm::yaml::SequenceTraits on T and implement two methods:
643 “size_t size(IO &io, T&)” and “T::value_type& element(IO &io, T&, size_t indx)”.
649 struct SequenceTraits<MySeq> {
650 static size_t size(IO &io, MySeq &list) { ... }
651 static MySeqEl element(IO &io, MySeq &list, size_t index) { ... }
654 The size() method returns how many elements are currently in your sequence.
655 The element() method returns a reference to the i'th element in the sequence.
656 When parsing YAML, the element() method may be called with an index one bigger
657 than the current size. Your element() method should allocate space for one
658 more element (using default constructor if element is a C++ object) and returns
659 a reference to that new allocated space.
664 A YAML "flow sequence" is a sequence that when written to YAML it uses the
665 inline notation (e.g [ foo, bar ] ). To specify that a sequence type should
666 be written in YAML as a flow sequence, your SequenceTraits specialization should
667 add "static const bool flow = true;". For instance:
672 struct SequenceTraits<MyList> {
673 static size_t size(IO &io, MyList &list) { ... }
674 static MyListEl element(IO &io, MyList &list, size_t index) { ... }
676 // The existence of this member causes YAML I/O to use a flow sequence
677 static const bool flow = true;
680 With the above, if you used MyList as the data type in your native data
681 strucutures, then then when converted to YAML, a flow sequence of integers
682 will be used (e.g. [ 10, -3, 4 ]).
687 Since a common source of sequences is std::vector<>, YAML I/O provids macros:
688 LLVM_YAML_IS_SEQUENCE_VECTOR() and LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR() which
689 can be used to easily specify SequenceTraits<> on a std::vector type. YAML
690 I/O does not partial specialize SequenceTraits on std::vector<> because that
691 would force all vectors to be sequences. An example use of the macros:
695 std::vector<MyType1>;
696 std::vector<MyType2>;
697 LLVM_YAML_IS_SEQUENCE_VECTOR(MyType1)
698 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(MyType2)
705 YAML allows you to define multiple "documents" in a single YAML file. Each
706 new document starts with a left aligned "---" token. The end of all documents
707 is denoted with a left aligned "..." token. Many users of YAML will never
708 have need for multiple documents. The top level node in their YAML schema
709 will be a mapping or sequence. For those cases, the following is not needed.
710 But for cases where you do want multiple documents, you can specify a
711 trait for you document list type. The trait has the same methods as
712 SequenceTraits but is named DocumentListTraits. For example:
717 struct DocumentListTraits<MyDocList> {
718 static size_t size(IO &io, MyDocList &list) { ... }
719 static MyDocType element(IO &io, MyDocList &list, size_t index) { ... }
725 When an llvm::yaml::Input or llvm::yaml::Output object is created their
726 constructors take an optional "context" parameter. This is a pointer to
727 whatever state information you might need.
729 For instance, in a previous example we showed how the conversion type for a
730 flags field could be determined at runtime based on the value of another field
731 in the mapping. But what if an inner mapping needs to know some field value
732 of an outer mapping? That is where the "context" parameter comes in. You
733 can set values in the context in the outer map's mapping() method and
734 retrieve those values in the inner map's mapping() method.
736 The context value is just a void*. All your traits which use the context
737 and operate on your native data types, need to agree what the context value
738 actually is. It could be a pointer to an object or struct which your various
739 traits use to shared context sensitive information.
745 The llvm::yaml::Output class is used to generate a YAML document from your
746 in-memory data structures, using traits defined on your data types.
747 To instantiate an Output object you need an llvm::raw_ostream, and optionally
752 class Output : public IO {
754 Output(llvm::raw_ostream &, void *context=NULL);
756 Once you have an Output object, you can use the C++ stream operator on it
757 to write your native data as YAML. One thing to recall is that a YAML file
758 can contain multiple "documents". If the top level data structure you are
759 streaming as YAML is a mapping, scalar, or sequence, then Output assumes you
760 are generating one document and wraps the mapping output
761 with "``---``" and trailing "``...``".
765 using llvm::yaml::Output;
767 void dumpMyMapDoc(const MyMapType &info) {
768 Output yout(llvm::outs());
772 The above could produce output like:
781 On the other hand, if the top level data structure you are streaming as YAML
782 has a DocumentListTraits specialization, then Output walks through each element
783 of your DocumentList and generates a "---" before the start of each element
784 and ends with a "...".
788 using llvm::yaml::Output;
790 void dumpMyMapDoc(const MyDocListType &docList) {
791 Output yout(llvm::outs());
795 The above could produce output like:
810 The llvm::yaml::Input class is used to parse YAML document(s) into your native
811 data structures. To instantiate an Input
812 object you need a StringRef to the entire YAML file, and optionally a context
817 class Input : public IO {
819 Input(StringRef inputContent, void *context=NULL);
821 Once you have an Input object, you can use the C++ stream operator to read
822 the document(s). If you expect there might be multiple YAML documents in
823 one file, you'll need to specialize DocumentListTraits on a list of your
824 document type and stream in that document list type. Otherwise you can
825 just stream in the document type. Also, you can check if there was
826 any syntax errors in the YAML be calling the error() method on the Input
831 // Reading a single document
832 using llvm::yaml::Input;
834 Input yin(mb.getBuffer());
836 // Parse the YAML file
847 // Reading multiple documents in one file
848 using llvm::yaml::Input;
850 LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(std::vector<MyDocType>)
852 Input yin(mb.getBuffer());
854 // Parse the YAML file
855 std::vector<MyDocType> theDocList;