rapidyaml 0.16.0
parse and emit YAML, and do it fast
Loading...
Searching...
No Matches
Ints Event Handler
Parse utilities » Event Handlers

An event handler used by ParseEngine to creates an integer buffer with a very compact representation of the YAML tree in a source buffer. More...

Namespaces

namespace  c4::yml::extra::ievt

Classes

struct  c4::yml::extra::EventHandlerInts
 A parser event handler that creates a compact representation of the YAML tree in a contiguous buffer of integers. More...

Typedefs

using c4::yml::extra::evt_size = int32_t
 data type for integer events size.

Functions

evt_size c4::yml::extra::estimate_events_ints_size (csubstr src)
 Read YAML source and, without undergoing a full parse, estimate the size of the integer buffer required for EventHandlerInts.
size_t c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz, substr evts_testsuite)
 Create a testsuite event string from integer events.
template<class Container>
void c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz, Container *evts_testsuite)
 Create a testsuite event string from integer events, writing into an output container.
template<class Container>
Container c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
 Create a testsuite event string from integer events, returning a new container with the result.
void c4::yml::extra::events_ints_print (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
 Print integer events to stdout.

Detailed Description

An event handler used by ParseEngine to creates an integer buffer with a very compact representation of the YAML tree in a source buffer.

This feature is an extra and is not part of the main rapidyaml library.

This is meant for use by other programming languages, and supports container keys (unlike the ryml tree). It parses faster than the ryml tree parser, because the resulting data structure is much simpler.

The resulting integer buffer is a linear array of integers containing encoded YAML events (as a mask of ievt::EventBits), which in some cases (eg scalars) are followed by an encoded string (encoded as an offset and length to the parsed source buffer).

For example, parsing [a, bb, ccc] results in the following event buffer (grouped to highlight the event sequence structure):

using namespace c4::yml::extra::ievt;
const evt_bits arr[] = { // result of parsing: [a, bb, ccc]
BSTR, // begin stream
BDOC, // begin doc
VAL_|BSEQ|FLOW, // begin seq as val, flow
VAL_|SCLR|PLAI, 1, 1, // val scalar, plain style, "a" : string starts at offset 1 and has length 1
VAL_|SCLR|PLAI|PSTR, 4, 2, // val scalar, plain style, "bb" : string starts at offset 4 and has length 2; preceded by a string event (PSTR)
VAL_|SCLR|PLAI|PSTR, 8, 3, // val scalar, plain style, "ccc": string starts at offset 8 and has length 3; preceded by a string event (PSTR)
ESEQ|PSTR, // end seq; preceded by a string event (PSTR)
EDOC, // end doc
ESTR, // end stream
};
c4::yml::extra::ievt::evt_bits
int32_t evt_bits
data type for integer events bits.
Definition event_handler_ints.hpp:183
c4::yml::extra::ievt::PSTR
@ PSTR
Special flag to enable look-back in the event array. It signifies that the previous event has a strin...
Definition event_handler_ints.hpp:258
c4::yml::extra::ievt::SCLR
@ SCLR
scalar (=VAL in test suite events)
Definition event_handler_ints.hpp:214
c4::yml::extra::ievt::ESTR
@ ESTR
end stream (-STR in test suite events)
Definition event_handler_ints.hpp:209
c4::yml::extra::ievt::BSTR
@ BSTR
begin stream (+STR in test suite events)
Definition event_handler_ints.hpp:208
c4::yml::extra::ievt::BSEQ
@ BSEQ
begin seq (+SEQ in test suite events)
Definition event_handler_ints.hpp:204
c4::yml::extra::ievt::ESEQ
@ ESEQ
end seq (-SEQ in test suite events)
Definition event_handler_ints.hpp:205
c4::yml::extra::ievt::FLOW
@ FLOW
container: flow: [] for seqs or {} for maps
Definition event_handler_ints.hpp:229
c4::yml::extra::ievt::BDOC
@ BDOC
begin doc (+DOC in test suite events)
Definition event_handler_ints.hpp:210
c4::yml::extra::ievt::EDOC
@ EDOC
end doc (-DOC in test suite events)
Definition event_handler_ints.hpp:211

Here is a sketch clarifying the meaning of this event sequence:

source : [a, bb, ccc]
has a string........
| offset "a"
| | length "a"
| | |
event0 event1 event2 [ event3 "a"......|..|
| | | | | |
(start) +--------+-------+------------------+---------------+--+-----> (continued)
arr[i] : BSTR BDOC VAL_|BSEQ|FLOW VAL_|SCLR|PLAI..1..1
i : 0 1 2 3 4 5
has a string............. has a string.............
| offset "bb" | offset "ccc"
| | length "bb" | | length "ccc"
| | | | | |
event4 "bb"..........|..| event5 "ccc".........|..|
| | | | | |
(cont)--> -----+--------------------+--+--------------+--------------------+--+-----> (continued)
arr[i] : VAL_|SCLR|PLAI|PSTR..4..2 VAL_|SCLR|PLAI|PSTR..8..3
i : 6 | 7 8 9 | 10 11
| |
prev event has string prev event has string
(to get to prev, jump (to get to prev, jump
back 3 slots: ie 6->3) back 3 slots: ie 9->6)
event6 ] event7 event8
| | |
(cont)--> -----+-------------+--------+-----| (end)
arr[i] : ESEQ|PSTR EDOC ESTR
i : 12 | 13 14
|
prev event has string
(to get to it, jump
back 3 slots: ie 12->9)

Note that the buffer contains both events and strings encoded as integer pairs. That is, events that have an associated string are immediately followed by two integers providing the offset and length of that string in the source buffer. (In the example above, this happens in the events for the strings a, bb, and ccc at positions 3, 6 and 9, respectively).

The flag ievt::PSTR and the mask ievt::WSTR are provided to enable easier iteration over the array: you can use them to test for presence of a string when iterating over the array.

The flag ievt::PSTR announces that an event is preceded by a string. That is, the previous event has a string, so that when this flag is found while iterating right-to-left, a jump of -3 should be used to get at the bitmask of the previous event. (In the example above, this flag is present for the events for bb and ccc, but not a because it is not preceded by a string).

Likewise, to signify that the current event is followed by a string, there is the mask ievt::WSTR, which is a mask of all the flags of events that have a string: ievt::SCLR, ievt::ALIA, ievt::ANCH and ievt::TAG_. While iterating left-to-right in the array, presence of any of the bits in the mask ievt::WSTR means that a jump of +3 should be employed to get at the bitmask of the next event.

Here's another example with the result of parsing a: bb

const evt_bits arr[] = { // result of parsing: `a: bb`
BSTR, // begin stream
BDOC, // begin doc
VAL_|BMAP|BLCK, // begin map as val, block
KEY_|SCLR|PLAI, 0, 1, // key scalar, plain style: "a" starts at offset 0 and has length 1
VAL_|SCLR|PLAI|PSTR, 3, 2, // val scalar, plain style: "bb" starts at offset 3 and has length 2
EMAP|PSTR, // end map
EDOC, // end doc
ESTR, // end stream
};
c4::yml::extra::ievt::EMAP
@ EMAP
end map (-MAP in test suite events)
Definition event_handler_ints.hpp:207
c4::yml::extra::ievt::BMAP
@ BMAP
begin map (+MAP in test suite events)
Definition event_handler_ints.hpp:206

Typical code to iterate left-to-right over the array will look like this:

// source buffer, modified in place during parsing (IMPORTANT!)
substr src = ...;
substr arena = ...; // arena used for scalars/tags that are extended during filtering
// events resulting from parsing
const int events[] = {...};
int events_size = ...;
for(int i = 0; i < events_size; ++i)
{
if(events[i] & ievt::WSTR) // this event has a string following it
{
size_t offset = (size_t)events[i+1];
size_t length = (size_t)events[i+2];
csubstr region = (events[i] & ievt::AREN) ? arena : src; // is the string in the arena?
csubstr str = region.sub(offset, length); // get the string
...
i += 2; // skip the two ints of the string
// (the jump is three places; the loop adds the other place)
}
else // this is a single-int event
{
...
}
}
c4::substr
basic_substring< char > substr
a mutable string view
Definition substr.hpp:2355
c4::csubstr
basic_substring< const char > csubstr
an immutable string view
Definition substr.hpp:2356
c4::yml::extra::ievt::WSTR
@ WSTR
WithSTRing: mask of all events that encode a string following the event. For such events,...
Definition event_handler_ints.hpp:251
c4::yml::extra::ievt::AREN
@ AREN
Special flag to mark events whose string was placed in the arena. This happens when the filtered stri...
Definition event_handler_ints.hpp:246
c4::basic_substring::sub
basic_substring sub(size_t first) const noexcept
return [first,len[
Definition substr.hpp:502

Typedef Documentation

◆ evt_size

using c4::yml::extra::evt_size = int32_t

data type for integer events size.

This is set to an int32_t integer to allow compatibility with a wide range of processing languages.

Definition at line 176 of file event_handler_ints.hpp.

Function Documentation

◆ estimate_events_ints_size()

evt_size c4::yml::extra::estimate_events_ints_size ( csubstr src)

Read YAML source and, without undergoing a full parse, estimate the size of the integer buffer required for EventHandlerInts.

This estimation is meant to exceed the actual number of required events.

Note
This function must overpredict. It does so for every case in the hundreds/thousands of extensive tests of rapidyaml – both for the YAML test suite and the internal cases. If you find a case where that does not hold, it is a bug. Please report it at https://github.com/biojppm/rapidyaml/issues!

Definition at line 25 of file event_handler_ints.cpp.

26{
27 evt_size count = 7; // BSTR + BDOC + =VAL + EDOC + ESTR
28 for(size_t i = 0; i < src.len; ++i)
29 {
30 switch(src.str[i])
31 {
32 case ':': // this has strings preceding/following it
33 case ',': // overestimate, assume map
34 case '%': // assume TAGD->string + TAGV->string
35 count += 6;
36 break;
37 // these have (or are likely to have) a string following it
38 case '-':
39 case '&':
40 case '*':
41 case '<':
42 case '!':
43 case '\'':
44 case '"':
45 case '|':
46 case '>':
47 case '\n':
48 count += 3;
49 break;
50 case '[':
51 case ']':
52 count += 4;
53 break;
54 case '{':
55 case '}':
56 count += 7;
57 break;
58 case '?':
59 count += 5;
60 break;
61 }
62 }
63 return count;
64}
c4::yml::extra::evt_size
int32_t evt_size
data type for integer events size.
Definition event_handler_ints.hpp:176
c4::basic_substring::len
size_t len
the length of the substring
Definition substr.hpp:218
c4::basic_substring::str
C * str
a restricted pointer to the first character of the substring
Definition substr.hpp:216

Referenced by estimate_events_ints_size().

◆ events_ints_to_testsuite() [1/3]

size_t c4::yml::extra::events_ints_to_testsuite ( csubstr parsed_yaml,
csubstr arena,
ievt::evt_bits const * evts_ints,
ievt::evt_bits evts_ints_sz,
substr evts_testsuite )

Create a testsuite event string from integer events.

This overload receives a buffer where the string is to be written, and returns the size needed to accomodate the result. The size of the buffer is strictly respected. The caller must check that the returned size is smaller than the buffer's size to ensure that the result is complete. If that's not the case, the user must resize the buffer and call again.

Definition at line 33 of file ints_to_testsuite.cpp.

38{
39 auto getstr = [&](ievt::evt_bits i){
40 bool in_arena = evts_ints[i] & ievt::AREN;
41 csubstr region = !in_arena ? parsed_yaml : arena;
42 return region.sub((size_t)evts_ints[i+1], (size_t)evts_ints[i+2]);
43 };
44 size_t sz = 0;
45 auto append = [&](csubstr s){
46 size_t next = sz + s.len;
47 if (s.len && (next <= evts_test_suite.len && evts_test_suite.len))
48 memcpy(evts_test_suite.str + sz, s.str, s.len);
49 sz = next;
50 };
51 bool has_tag = false;
52 csubstr tag;
53 auto maybe_append_tag = [&]{
54 if(has_tag)
55 {
56 if(tag.begins_with('<'))
57 {
58 append(" ");
59 append(tag);
60 }
61 else
62 {
63 RYML_ASSERT_BASIC_(tag.begins_with('!'));
64 append(" <");
65 append(tag);
66 append(">");
67 }
68 }
69 has_tag = false;
70 };
71 bool has_anchor = false;
72 csubstr anchor;
73 auto maybe_append_anchor = [&]{
74 if(has_anchor)
75 {
76 append(" &");
77 append(anchor);
78 }
79 has_anchor = false;
80 };
81 auto append_cont = [&](csubstr evt, csubstr style){
82 append(evt);
83 if(style.len)
84 {
85 append(" ");
86 append(style);
87 }
88 maybe_append_anchor();
89 maybe_append_tag();
90 append("\n");
91 };
92 auto append_esc = [&](csubstr str){
93 substr buf = sz <= evts_test_suite.len ? evts_test_suite.sub(sz) : evts_test_suite.last(0);
94 sz += escape_scalar(buf, str);
95 append("\n");
96 };
97 auto append_val = [&](csubstr evt, csubstr val){
98 append("=VAL");
99 maybe_append_anchor();
100 maybe_append_tag();
101 append(" ");
102 append(evt);
103 append_esc(val);
104 };
105 ievt::evt_bits evt = 0;
106 for(ievt::evt_bits i = 0; i < evts_ints_sz; i += (evt & ievt::WSTR) ? 3 : 1)
107 {
108 evt = evts_ints[i];
109 if(evt & ievt::SCLR)
110 {
111 csubstr s = getstr(i);
112 if(evt & ievt::SQUO)
113 append_val("'", s);
114 else if(evt & ievt::DQUO)
115 append_val("\"", s);
116 else if(evt & ievt::LITL)
117 append_val("|", s);
118 else if(evt & ievt::FOLD)
119 append_val(">", s);
120 else //if(evt & ievt::PLAI)
121 append_val(":", s);
122 }
123 else if((evt & ievt::BSEQ) == ievt::BSEQ)
124 {
125 if(evt & ievt::FLOW)
126 append_cont("+SEQ", "[]");
127 else
128 append_cont("+SEQ", "");
129 }
130 else if((evt & ievt::ESEQ) == ievt::ESEQ)
131 {
132 append("-SEQ\n");
133 }
134 else if((evt & ievt::BMAP) == ievt::BMAP)
135 {
136 if(evt & ievt::FLOW)
137 append_cont("+MAP", "{}");
138 else
139 append_cont("+MAP", "");
140 }
141 else if((evt & ievt::EMAP) == ievt::EMAP)
142 {
143 append("-MAP\n");
144 }
145 else if(evt & ievt::ALIA)
146 {
147 append("=ALI *");
148 append(getstr(i));
149 append("\n");
150 }
151 else if(evt & ievt::TAG_)
152 {
153 has_tag = true;
154 tag = getstr(i);
155 }
156 else if(evt & ievt::ANCH)
157 {
158 has_anchor = true;
159 anchor = getstr(i);
160 }
161 else if((evt & ievt::BDOC) == ievt::BDOC)
162 {
163 if(evt & ievt::EXPL)
164 append("+DOC ---\n");
165 else
166 append("+DOC\n");
167 }
168 else if((evt & ievt::EDOC) == ievt::EDOC)
169 {
170 if(evt & ievt::EXPL)
171 append("-DOC ...\n");
172 else
173 append("-DOC\n");
174 }
175 else if((evt & ievt::BSTR) == ievt::BSTR)
176 {
177 append("+STR\n");
178 }
179 else if((evt & ievt::ESTR) == ievt::ESTR)
180 {
181 append("-STR\n");
182 }
183 }
184 return sz;
185}
c4::yml::extra::ievt::LITL
@ LITL
scalar: block literal (|)
Definition event_handler_ints.hpp:227
c4::yml::extra::ievt::DQUO
@ DQUO
scalar: double-quoted ("")
Definition event_handler_ints.hpp:226
c4::yml::extra::ievt::FOLD
@ FOLD
scalar: block folded (>)
Definition event_handler_ints.hpp:228
c4::yml::extra::ievt::EXPL
@ EXPL
--- (with BDOC) or ... (with EDOC)
Definition event_handler_ints.hpp:202
c4::yml::extra::ievt::SQUO
@ SQUO
scalar: single-quoted (')
Definition event_handler_ints.hpp:225
c4::yml::escape_scalar
size_t escape_scalar(substr buffer, csubstr scalar, bool keep_newlines=false)
Escape a scalar to an existing buffer, using escape_scalar_fn.
Definition escape_scalar.hpp:241
c4::basic_substring::begins_with
bool begins_with(const C c) const noexcept
true if the first character of the string is c
Definition substr.hpp:850

Referenced by events_ints_to_testsuite(), events_ints_to_testsuite(), and events_ints_to_testsuite().

◆ events_ints_to_testsuite() [2/3]

template<class Container>
void c4::yml::extra::events_ints_to_testsuite ( csubstr parsed_yaml,
csubstr arena,
ievt::evt_bits const * evts_ints,
ievt::evt_bits evts_ints_sz,
Container * evts_testsuite )

Create a testsuite event string from integer events, writing into an output container.

Definition at line 39 of file ints_to_testsuite.hpp.

44{
45 size_t len = events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, to_substr(*evts_testsuite));
46 if(len > evts_testsuite->size())
47 {
48 evts_testsuite->resize(len);
49 len = events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, to_substr(*evts_testsuite));
50 }
51 evts_testsuite->resize(len);
52}
c4::yml::extra::events_ints_to_testsuite
size_t events_ints_to_testsuite(csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz, substr evts_test_suite)
Create a testsuite event string from integer events.
Definition ints_to_testsuite.cpp:33
c4::to_substr
substr to_substr(char(&s)[N]) noexcept
Definition substr.hpp:2376

◆ events_ints_to_testsuite() [3/3]

template<class Container>
Container c4::yml::extra::events_ints_to_testsuite ( csubstr parsed_yaml,
csubstr arena,
ievt::evt_bits const * evts_ints,
ievt::evt_bits evts_ints_sz )

Create a testsuite event string from integer events, returning a new container with the result.

Definition at line 58 of file ints_to_testsuite.hpp.

62{
63 Container ret;
64 events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, &ret);
65 return ret;
66} // LCOV_EXCL_LINE

◆ events_ints_print()

void c4::yml::extra::events_ints_print ( csubstr parsed_yaml,
csubstr arena,
ievt::evt_bits const * evts,
ievt::evt_bits evts_sz )

Print integer events to stdout.

Definition at line 103 of file ints_utils.cpp.

104{
105 char buf[200];
106 for(ievt::evt_bits evtpos = 0, evtnumber = 0;
107 evtpos < evts_sz;
108 ++evtnumber,
109 evtpos += ((evts[evtpos] & ievt::WSTR) ? 3 : 1))
110 {
111 ievt::evt_bits evt = evts[evtpos];
112 csubstr flags = ievt::to_str_sub(buf, evt);
113 printf("[%d][%d] %.*s(0x%x)", evtnumber, evtpos, (int)flags.len, flags.str, evt);
114 if (evt & ievt::WSTR)
115 {
116 bool in_arena = evt & ievt::AREN;
117 csubstr region = !in_arena ? parsed_yaml : arena;
118 bool safe = (evts[evtpos + 1] >= 0)
119 && (evts[evtpos + 2] >= 0)
120 && (evts[evtpos + 1] <= (ievt::evt_bits)region.len) // NOLINT
121 && (evts[evtpos + 2] <= ((ievt::evt_bits)region.len - evts[evtpos + 1]));
122 const char *str = safe ? (region.str + evts[evtpos + 1]) : "ERR!!!";
123 ievt::evt_bits len = safe ? evts[evtpos + 2] : 6;
124 printf(": %d [%d]~~~%.*s~~~", evts[evtpos+1], evts[evtpos+2], len, str);
125 if(in_arena)
126 printf(" (arenasz=%zu)", arena.len);
127 else
128 printf(" (srcsz=%zu)", parsed_yaml.len);
129 }
130 printf("\n");
131 }
132}
c4::yml::extra::ievt::to_str_sub
csubstr to_str_sub(substr buf, ievt::evt_bits flags)
Convert bit mask of ievt::EventBits to text.
Definition ints_utils.cpp:80