granit-parser - crates.io Package Compare versions

+188

COMMENTS.md

		# Comment Support Plan

		This document tracks work needed to preserve YAML comments as parser events with
		normal parser spans. This is a breaking event-stream change. When implementing
		changes, always add relevant unit tests, format all code at the end and ensure
		pedantic Clippy is passing. All public functions and classes must have
		documentation comments.

		Once you implement the step, make edit in this document marking it as done.

		## Goals

		- Preserve comments as presentation data in the parser event stream.
		- Emit comments as parser events with the existing `(Event, Span)` result shape.
		- Keep comment spans local to the source parsed by the parser that emitted them.
		- Preserve current scanner/parser error behavior.
		- Support both `StrInput` and streaming `BufferedInput`.

		## Non-goals

		- Do not make comments part of the semantic YAML tree.
		- Do not alter scalar spans to include trailing comments.
		- Do not add source IDs, file names, or include provenance to `Span` or `Event`.
		- Do not add a separate `ParserStack` comment collection API for the event-based
		implementation.

		## Event Semantics

		- Comments are emitted in source order as parser events.
		- A comment event uses the same spanned parser output as other events:
		`(Event::Comment(text, placement), span)`.
		- The comment span covers the whole source comment, including `#` and excluding
		the line break.
		- The comment event payload is the raw text exactly after `#`, excluding only the
		line break. Preserve leading spaces, including one space immediately after `#`.
		- The comment event placement is presentation metadata that provides a best-effort
		positional hint such as `Above`, `Right`, `Free`, or `Last`.
		- Comment events are presentation metadata. Consumers building YAML data trees
		must ignore them.
		- Comment events do not advance the YAML grammar state; after a comment is
		emitted, parsing resumes in the same parser state.
		- `ParserStack` and `ReplayParser` should preserve included-document comments by
		carrying them through the normal event stream.
		- Included-document comment spans remain local to the included document/source,
		just as other included-document event spans do today. Source/file provenance is
		left to callers that layer names around `ParserStack`.

		## Superseded Collection Design

		- The earlier `comments()` / `take_comments()` collection design is superseded.
		- Do not preserve comment storage as a second public API after `Event::Comment`
		is implemented.
		- Remove `Parser::with_comments()`, `Parser::comments()`,
		`Parser::take_comments()`, `Scanner::with_comments()`, `Scanner::comments()`,
		and `Scanner::take_comments()`.
		- Remove scanner-side comment collection fields (`comments`, `collect_comments`)
		unless they are temporarily needed during migration.
		- Replace collection tests with event-stream tests instead of maintaining both
		behaviors.

		## Task List

		### API Design

		- [x] Add `Event::Comment(Cow<'input, str>, Placement)`.
		- [x] Store the raw comment payload exactly after `#`, excluding only the line break.
		- [x] Preserve leading spaces in the payload, including a single space immediately after `#`.
		- [x] Store a best-effort placement hint for correlating presentation comments.
		- [x] Use the companion parser `Span` for the full source comment range.
		- [x] Add `TokenType::Comment(Comment<'input>)` or an equivalent internal scanner token.
		- [x] Store the comment payload in the token.
		- [x] Store the full source comment range in the token span.
		- [x] Store the scanner's initial placement hint.
		- [x] Decide whether the public `Comment<'input>` type remains useful after the event-based API.
		- [x] If kept, document it as a convenience struct rather than the primary parser API.
		- [x] If removed, migrate tests and docs to `Event::Comment` plus `Span`. Not applicable:
		`Comment` is kept as a convenience struct.
		- [x] Remove the superseded opt-in collection APIs.
		- [x] Remove `Parser::with_comments()`, `Parser::comments()`, and `Parser::take_comments()`.
		- [x] Remove `Scanner::with_comments()`, `Scanner::comments()`, and `Scanner::take_comments()`.
		- [x] Remove collection-semantics documentation once the event API is complete.

		### Scanner Capture And Tokenization

		- [x] Rework scanner comment capture to emit comment tokens instead of only collecting
		comments in side storage.
		- [x] Capture comments through `skip_to_next_token`, `skip_yaml_whitespace`, and
		`skip_ws_to_eol` without losing the current error behavior.
		- [x] Preserve the current scanner error behavior; comment tokens are part of the
		normal token stream after this change.
		- [x] Keep `#` inside quoted scalars and block scalar content out of the comment stream.
		- [x] Preserve zero-copy comment payloads for `StrInput` where possible.
		- [x] Preserve owned comment payloads for `BufferedInput`.
		- [x] Keep marker accounting identical to normal skipped-comment behavior.
		- [x] Remove scanner-side comment collection storage once comment tokens are emitted.

		### Discard Points To Rework

		- [x] Rework `Scanner::skip_to_next_token`.
		- [x] Emit full-line and inter-token comment tokens.
		- [x] Preserve line break consumption and simple-key behavior.
		- [x] Rework `Scanner::skip_yaml_whitespace`.
		- [x] Emit comments after explicit key whitespace.
		- [x] Preserve the current `expected whitespace` behavior.
		- [x] Rework `Scanner::skip_ws_to_eol`.
		- [x] Emit comment tokens before end-of-line comments are discarded.
		- [x] Preserve `SkipTabs` results.
		- [x] Preserve the existing error for comments not separated from tokens by whitespace.
		- [x] Detect the unseparated-comment error before comment token emission.
		- [x] Do not consume or emit an unseparated `#` as a valid comment.
		- [x] Review all callers of `skip_ws_to_eol`.
		- [x] Directives.
		- [x] Document end marker handling.
		- [x] Flow collection start/end.
		- [x] Flow entries.
		- [x] Block entries.
		- [x] Block scalar headers.
		- [x] Quoted scalars after the closing quote.
		- [x] Plain scalar tab handling.
		- [x] Mapping values after `:`.

		### Parser Integration

		- [x] Emit `Event::Comment` from `Parser::next_event_impl` before normal YAML
		grammar-state handling.
		- [x] Consume exactly one comment token per emitted comment event.
		- [x] Leave the parser state unchanged after emitting a comment event.
		- [x] Ensure comments after `StreamStart`, around document markers, inside flow
		collections, and before `StreamEnd` are emitted in source order.
		- [x] Keep the parser result shape unchanged: `Result<(Event<'input>, Span), ScanError>`.
		- [x] Update `peek`, `next_event`, `load`, `try_load`, and iterator behavior to
		carry comment events consistently.
		- [x] Preserve comments in `ReplayParser` without a side channel.
		- [x] Replayed event streams should store comment events in the existing event vector.
		- [x] `ReplayParser::load` and `try_load` should forward comment events like all other events.
		- [x] Preserve comments in included documents through `ParserStack`.
		- [x] `ParserStack::resolve` should preserve comments because includes are replayed
		through the normal parser event stream.
		- [x] Included-document comment spans remain local to the included source.
		- [x] Do not add source IDs, file names, or a dedicated comment collection API to
		`ParserStack` for this implementation.

		### Tests

		- [x] Add parser event tests for `Event::Comment`.
		- [x] Full-line comments are emitted as events with spans.
		- [x] Indented full-line comments are emitted as events with spans.
		- [x] Trailing comments after plain scalars are emitted after the scalar event.
		- [x] Empty-ish comments `#` and `# ` preserve payload text.
		- [x] CRLF comment spans end before `\r`, not after `\n`.
		- [x] `peek()` returns and preserves a pending comment event.
		- [x] `load` and `try_load` deliver comment events to receivers.
		- [x] Add parser-stack and replay tests for comment events.
		- [x] `ReplayParser` preserves comment events.
		- [x] `ParserStack` forwards comment events from stacked parsers.
		- [x] `ParserStack::resolve` / `push_include` forwards comments from included documents.
		- [x] Included-document comment spans remain local to the included source.
		- [x] Update existing parser-level collection tests.
		- [x] Replace "event stream unchanged when comments are enabled" expectations with
		explicit `Event::Comment` expectations.
		- [x] Remove `comments()` and `take_comments()` drain tests.
		- [x] Migrate scanner collection tests to scanner-token or parser-event tests.
		- [x] Full-line comments.
		- [x] Indented full-line comments.
		- [x] Trailing comments after plain scalars.
		- [x] Multiple consecutive comment lines.
		- [x] EOF immediately after a comment.
		- [x] Empty-ish comment: `#`.
		- [x] Empty-ish comment with one payload space: `# `.
		- [x] CRLF comment spans end before `\r`, not after `\n`.
		- [x] Comments after syntax elements: directives, document markers, tags,
		anchors, aliases, flow delimiters, flow entries, block entries, block
		scalar headers, quoted scalars, plain scalars, and mapping values.
		- [x] Negative cases: `#` inside quoted scalars and block scalar content,
		unseparated comments, and BS4K comment-interrupted multiline plain scalar.
		- [x] Non-ASCII payloads, character offsets, byte offsets for `StrInput`, and
		matching behavior between `StrInput` and `BufferedInput`.

		### Documentation
		- [x] Add a crate-level example in `src/lib.rs`.
		- [x] Explain that `Event::Comment` is presentation metadata, not YAML data.
		- [x] Document that comment locations use the normal companion `Span`.
		- [x] Document that non-spanned receivers receive `Event::Comment(text, placement)`,
		while spanned receivers receive the comment span in `on_event`.
		- [x] Document `span.slice(source)` behavior for comments.
		- [x] Document that included-document comment spans are local to the included source,
		matching existing included-document event spans.
		- [x] Update `CHANGELOG.md` when the implementation is complete.

+29

examples/custom_tags.rs

		use granit_parser::{Event, Parser};

		fn main() -> Result<(), Box<dyn std::error::Error>> {
		let yaml = "opw_kinematics_joint_offsets: !degrees [0, 0, -90, 0, 0, 180]\n";

		for next in Parser::new_from_str(yaml) {
		let (event, span) = next?;

		let node_kind = match &event {
		Event::Scalar(..) => "scalar",
		Event::SequenceStart(..) => "sequence",
		Event::MappingStart(..) => "mapping",
		_ => continue,
		};

		if let Some(tag) = event.tag() {
		let (handle, suffix) = tag.parts();
		println!(
		"tag={tag} custom={} parts=({handle:?}, {suffix:?}) node={node_kind} event={event:?} chars={}..{} bytes={:?}",
		tag.is_custom(),
		span.start.index(),
		span.end.index(),
		span.byte_range()
		);
		}
		}

		Ok(())
		}

+63

examples/include_stack.rs

		use granit_parser::{
		parser_stack::ParserStack, Event, Parser, ParserTrait, ScanError, StrInput, Tag,
		};

		type IncludeStack = ParserStack<'static, core::iter::Empty<char>, StrInput<'static>>;

		const ROOT_YAML: &str = r"
		root:
		before: &root_anchor from-root
		include: !include something.yaml
		after: &after_anchor after-include
		";

		fn resolve_include(name: &str) -> Result<String, ScanError> {
		match name {
		"something.yaml" => Ok("included: &included_anchor from-include\n".to_string()),
		_ => Err(ScanError::new_str(
		granit_parser::Marker::new(0, 1, 0),
		"include not found",
		)),
		}
		}

		fn is_include_tag(tag: &Tag) -> bool {
		tag.parts() == ("!", "include")
		}

		fn main() -> Result<(), ScanError> {
		let mut stack: IncludeStack = ParserStack::new();
		stack.push_str_parser(Parser::new_from_str(ROOT_YAML), "root.yaml".to_string());
		stack.set_resolver(resolve_include);

		let mut anchored_scalars = Vec::new();

		while let Some(next) = stack.next_event() {
		let (event, _) = next?;

		match event {
		Event::Scalar(include_name, _, _, Some(tag)) if is_include_tag(tag.as_ref()) => {
		stack.push_include(include_name.as_ref())?;
		}
		Event::Scalar(value, _, anchor_id, _) if anchor_id > 0 => {
		anchored_scalars.push((value.into_owned(), anchor_id));
		}
		_ => {}
		}
		}

		assert_eq!(
		anchored_scalars,
		vec![
		("from-root".to_string(), 1),
		("from-include".to_string(), 2),
		("after-include".to_string(), 3),
		]
		);

		for (value, anchor_id) in anchored_scalars {
		println!("{value}: anchor {anchor_id}");
		}

		Ok(())
		}

+993

tests/comment.rs

		use std::borrow::Cow;

		use granit_parser::{
		BorrowedInput, BufferedInput, Comment, Event, EventReceiver, Marker, Parser, Placement,
		ScalarStyle, ScanError, Scanner, Span, StrInput, Token, TokenType, TryEventReceiver,
		};

		fn parser_events(source: &str) -> Result<Vec<(Event<'_>, Span)>, ScanError> {
		Parser::new_from_str(source).collect()
		}

		fn first_empty_scalar_span(events: &[(Event<'_>, Span)]) -> Span {
		events
		.iter()
		.find_map(\|(event, span)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) if value.as_ref() == "~" => Some(*span),
		_ => None,
		})
		.expect("empty scalar should be emitted")
		}

		fn assert_monotonic_spans(events: &[(Event<'_>, Span)]) {
		let mut last: Option<(&Event<'_>, Span)> = None;

		for (event, span) in events {
		if let Some((last_event, last_span)) = last {
		assert!(
		span.start.index() >= last_span.start.index()
		&& span.end.index() >= last_span.end.index(),
		"event {event:?}@{span:?} came before event {last_event:?}@{last_span:?}",
		);
		}

		last = Some((event, *span));
		}
		}

		#[test]
		fn comment_type_is_a_convenience_container() {
		let span = Span::new(
		Marker::new(0, 1, 0).with_byte_offset(Some(0)),
		Marker::new(11, 1, 11).with_byte_offset(Some(11)),
		);
		let comment = Comment::new(span, " payload ");

		assert_eq!(comment.span, span);
		assert_eq!(comment.text, " payload ");
		assert_eq!(comment.placement, Placement::Free);
		assert_eq!(comment.as_ref(), " payload ");
		assert_eq!(comment.trimmed_text(), "payload");
		}

		#[test]
		fn comment_type_preserves_single_space_payload() {
		let span = Span::new(
		Marker::new(0, 1, 0).with_byte_offset(Some(0)),
		Marker::new(2, 1, 2).with_byte_offset(Some(2)),
		);
		let comment = Comment::new(span, " ");

		assert_eq!(comment.text, " ");
		assert_eq!(comment.trimmed_text(), "");
		}

		#[test]
		fn event_comment_stores_raw_payload() {
		let event = Event::Comment(" payload ".into(), Placement::Free);

		assert_eq!(event, Event::Comment(" payload ".into(), Placement::Free));
		assert!(!event.is_node());
		assert_eq!(event.scalar(), None);
		assert_eq!(event.tag(), None);
		assert_eq!(event.anchor_id(), None);
		assert_eq!(event.alias_id(), None);
		}

		#[test]
		fn token_comment_uses_span_for_full_source_comment() {
		let yaml = "key: value # payload\r\n";
		let span = Span::new(
		Marker::new(11, 1, 11).with_byte_offset(Some(11)),
		Marker::new(20, 1, 20).with_byte_offset(Some(20)),
		);
		let token = Token(
		span,
		TokenType::Comment(Comment::new(span, " payload").with_placement(Placement::Right)),
		);

		assert_eq!(token.0.slice(yaml), Some("# payload"));
		assert!(matches!(token.1, TokenType::Comment(ref comment)
		if comment.text == " payload" && comment.placement == Placement::Right));
		}

		#[test]
		fn scanner_emits_comment_tokens_in_source_order() {
		let yaml = "# top\n # indented\nkey: value # trailing\n#eof";
		let tokens = Scanner::new(StrInput::new(yaml)).collect::<Vec<Token<'_>>>();

		let comments: Vec<_> = tokens
		.iter()
		.filter_map(\|Token(span, token)\| match token {
		TokenType::Comment(comment) => Some((comment.text.as_ref(), span.slice(yaml))),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![
		(" top", Some("# top")),
		(" indented", Some("# indented")),
		(" trailing", Some("# trailing")),
		("eof", Some("#eof")),
		]
		);
		}

		#[test]
		fn scanner_assigns_initial_comment_placements() {
		let yaml = "# own line\nkey: value # right\n";
		let tokens = Scanner::new(StrInput::new(yaml)).collect::<Vec<Token<'_>>>();

		let comments: Vec<_> = tokens
		.iter()
		.filter_map(\|Token(_, token)\| match token {
		TokenType::Comment(comment) => Some((comment.text.as_ref(), comment.placement)),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![(" own line", Placement::Free), (" right", Placement::Right)]
		);
		}

		#[test]
		fn scanner_marks_same_line_comments_after_syntax_as_right() {
		let cases = [
		("- # sequence entry\n", " sequence entry"),
		("[ # flow sequence\n]\n", " flow sequence"),
		("? # explicit key\n: value\n", " explicit key"),
		("--- # document start\n", " document start"),
		];

		for (yaml, expected_text) in cases {
		let comment = Scanner::new(StrInput::new(yaml))
		.find_map(\|Token(_, token)\| match token {
		TokenType::Comment(comment) => Some(comment),
		_ => None,
		})
		.expect("comment token should be emitted");

		assert_eq!(comment.text, expected_text, "{yaml:?}");
		assert_eq!(comment.placement, Placement::Right, "{yaml:?}");
		}
		}

		#[test]
		fn scanner_emits_trailing_comment_after_plain_scalar_token() {
		let tokens = Scanner::new(StrInput::new("key: value # trailing\n")).collect::<Vec<Token<'_>>>();

		let value_index = tokens
		.iter()
		.position(\|Token(_, token)\| {
		matches!(token, TokenType::Scalar(ScalarStyle::Plain, value) if value == "value")
		})
		.expect("plain scalar token should be emitted");
		let comment_index = tokens
		.iter()
		.position(\|Token(_, token)\| {
		matches!(token, TokenType::Comment(comment) if comment.text == " trailing")
		})
		.expect("comment token should be emitted");

		assert!(value_index < comment_index);
		}

		#[test]
		fn scanner_emits_comments_after_syntax_tokens() {
		struct Case<'input> {
		yaml: &'input str,
		syntax_matches: fn(&TokenType<'_>) -> bool,
		expected_comment: &'input str,
		}

		let cases = [
		Case {
		yaml: "%YAML 1.2 # directive\n---\n",
		syntax_matches: \|token\| matches!(token, TokenType::VersionDirective(1, 2)),
		expected_comment: " directive",
		},
		Case {
		yaml: "%TAG !e! tag:example.com,2020:app/ # tag directive\n---\n",
		syntax_matches: \|token\| {
		matches!(token, TokenType::TagDirective(handle, prefix)
		if handle == "!e!" && prefix == "tag:example.com,2020:app/")
		},
		expected_comment: " tag directive",
		},
		Case {
		yaml: "---\nkey: value\n... # document end\n",
		syntax_matches: \|token\| matches!(token, TokenType::DocumentEnd),
		expected_comment: " document end",
		},
		Case {
		yaml: "[ # flow start\n]\n",
		syntax_matches: \|token\| matches!(token, TokenType::FlowSequenceStart),
		expected_comment: " flow start",
		},
		Case {
		yaml: "[a] # flow end\n",
		syntax_matches: \|token\| matches!(token, TokenType::FlowSequenceEnd),
		expected_comment: " flow end",
		},
		Case {
		yaml: "{a: b} # flow mapping end\n",
		syntax_matches: \|token\| matches!(token, TokenType::FlowMappingEnd),
		expected_comment: " flow mapping end",
		},
		Case {
		yaml: "[a, # flow entry\nb]\n",
		syntax_matches: \|token\| matches!(token, TokenType::FlowEntry),
		expected_comment: " flow entry",
		},
		Case {
		yaml: "? # explicit key\n: value\n",
		syntax_matches: \|token\| matches!(token, TokenType::Key),
		expected_comment: " explicit key",
		},
		Case {
		yaml: "!local # tag\nvalue\n",
		syntax_matches: \|token\| {
		matches!(token, TokenType::Tag(handle, suffix)
		if handle == "!" && suffix == "local")
		},
		expected_comment: " tag",
		},
		Case {
		yaml: "&a # anchor\nvalue\n",
		syntax_matches: \|token\| matches!(token, TokenType::Anchor(anchor) if anchor == "a"),
		expected_comment: " anchor",
		},
		Case {
		yaml: "ref: *a # alias\n",
		syntax_matches: \|token\| matches!(token, TokenType::Alias(alias) if alias == "a"),
		expected_comment: " alias",
		},
		Case {
		yaml: "key: \"value\" # quoted\n",
		syntax_matches: \|token\| matches!(token, TokenType::Scalar(ScalarStyle::DoubleQuoted, value) if value == "value"),
		expected_comment: " quoted",
		},
		Case {
		yaml: "key:\t# mapping value\n nested: value\n",
		syntax_matches: \|token\| matches!(token, TokenType::Value),
		expected_comment: " mapping value",
		},
		];

		for case in cases {
		let tokens = Scanner::new(StrInput::new(case.yaml)).collect::<Vec<Token<'_>>>();
		let syntax_index = tokens
		.iter()
		.position(\|Token(_, token)\| (case.syntax_matches)(token))
		.expect("syntax token should be emitted");
		let comment_index = tokens
		.iter()
		.position(\|Token(_, token)\| {
		matches!(token, TokenType::Comment(comment) if comment.text == case.expected_comment)
		})
		.expect("comment token should be emitted");

		assert!(syntax_index < comment_index, "{:?}", case.yaml);
		}
		}

		#[test]
		fn scanner_emits_comments_after_block_scalar_headers() {
		let yaml = "key: \| # block scalar header\n body\n";
		let tokens = Scanner::new(StrInput::new(yaml)).collect::<Vec<Token<'_>>>();

		assert!(tokens.iter().any(
		\|Token(_, token)\| matches!(token, TokenType::Comment(comment)
		if comment.text == " block scalar header")
		));
		assert!(tokens.iter().any(\|Token(_, token)\| {
		matches!(token, TokenType::Scalar(ScalarStyle::Literal, value) if value == "body\n")
		}));
		}

		#[test]
		fn scanner_preserves_empty_comment_payloads() {
		let yaml = "#\n# \n";
		let comments: Vec<_> = Scanner::new(StrInput::new(yaml))
		.filter_map(\|Token(span, token)\| match token {
		TokenType::Comment(comment) => Some((
		comment.text.into_owned(),
		span.slice(yaml).map(str::to_owned),
		)),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![
		(String::new(), Some("#".into())),
		(" ".into(), Some("# ".into()))
		]
		);
		}

		#[test]
		fn scanner_comment_span_stops_before_crlf() {
		let yaml = "# crlf\r\nkey: value\n";
		let comment = Scanner::new(StrInput::new(yaml))
		.find_map(\|Token(span, token)\| match token {
		TokenType::Comment(comment) => Some((comment.text.into_owned(), span)),
		_ => None,
		})
		.expect("comment token should be emitted");

		assert_eq!(comment.0, " crlf");
		assert_eq!(comment.1.slice(yaml), Some("# crlf"));
		}

		#[test]
		fn scanner_preserves_non_ascii_comment_payload_offsets() {
		let yaml = "# ž🎵\n";
		let comment = Scanner::new(StrInput::new(yaml))
		.find_map(\|Token(span, token)\| match token {
		TokenType::Comment(comment) => Some((comment, span)),
		_ => None,
		})
		.expect("comment token should be emitted");

		assert_eq!(comment.0.text, " ž🎵");
		assert_eq!(comment.1.slice(yaml), Some("# ž🎵"));
		assert_eq!(comment.1.start.index(), 0);
		assert_eq!(comment.1.end.index(), 4);
		assert_eq!(comment.1.start.byte_offset(), Some(0));
		assert_eq!(comment.1.end.byte_offset(), Some(8));
		}

		#[test]
		fn scanner_comment_text_is_borrowed_for_str_input() {
		let comment = Scanner::new(StrInput::new("# borrowed\n"))
		.find_map(\|Token(_, token)\| match token {
		TokenType::Comment(comment) => Some(comment.text),
		_ => None,
		})
		.expect("comment token should be emitted");

		match comment {
		Cow::Borrowed(text) => assert_eq!(text, " borrowed"),
		Cow::Owned(text) => panic!("expected borrowed comment text, got {text:?}"),
		}
		}

		#[test]
		fn scanner_comment_text_is_owned_for_buffered_input() {
		let comment = Scanner::new(BufferedInput::new("# streamed\n".chars()))
		.find_map(\|Token(_, token)\| match token {
		TokenType::Comment(comment) => Some(comment.text),
		_ => None,
		})
		.expect("comment token should be emitted");

		match comment {
		Cow::Owned(text) => assert_eq!(text, " streamed"),
		Cow::Borrowed(text) => panic!("expected owned comment text, got {text:?}"),
		}
		}

		#[test]
		fn scanner_comment_tokens_match_between_str_and_buffered_input() {
		fn collect_comments<'input, T>(
		scanner: Scanner<'input, T>,
		) -> Vec<(String, Placement, usize, usize)>
		where
		T: BorrowedInput<'input>,
		{
		scanner
		.filter_map(\|Token(span, token)\| match token {
		TokenType::Comment(comment) => Some((
		comment.text.into_owned(),
		comment.placement,
		span.start.index(),
		span.end.index(),
		)),
		_ => None,
		})
		.collect()
		}

		let yaml = "# top\nkey: value # ž🎵\n#eof";

		assert_eq!(
		collect_comments(Scanner::new(StrInput::new(yaml))),
		collect_comments(Scanner::new(BufferedInput::new(yaml.chars())))
		);
		}

		#[test]
		fn scanner_does_not_emit_comments_from_quoted_or_block_scalar_content() {
		let cases = [
		(
		"key: \"# not a comment\"\n",
		ScalarStyle::DoubleQuoted,
		"# not a comment",
		),
		(
		"key: '# not a comment'\n",
		ScalarStyle::SingleQuoted,
		"# not a comment",
		),
		(
		"key: \|\n # not a comment\n",
		ScalarStyle::Literal,
		"# not a comment\n",
		),
		];

		for (yaml, style, expected_value) in cases {
		let tokens = Scanner::new(StrInput::new(yaml)).collect::<Vec<Token<'_>>>();

		assert!(
		!tokens
		.iter()
		.any(\|Token(_, token)\| matches!(token, TokenType::Comment(_))),
		"{yaml:?}"
		);
		assert!(
		tokens.iter().any(\|Token(_, token)\| {
		matches!(token, TokenType::Scalar(scalar_style, value)
		if *scalar_style == style && value == expected_value)
		}),
		"{yaml:?}"
		);
		}
		}

		#[test]
		fn scanner_does_not_emit_unseparated_comment_after_quoted_scalar_error() {
		let mut scanner = Scanner::new(StrInput::new("key: \"value\"#bad\n"));
		let mut saw_comment = false;

		let error = loop {
		match scanner.next_token() {
		Ok(Some(Token(_, TokenType::Comment(_)))) => saw_comment = true,
		Ok(Some(_)) => {}
		Ok(None) => panic!("expected scanner error"),
		Err(error) => break error,
		}
		};

		assert_eq!(
		error.info(),
		"comments must be separated from other tokens by whitespace"
		);
		assert!(!saw_comment);
		}

		#[test]
		fn scanner_treats_unseparated_hash_after_plain_scalar_as_content() {
		let tokens = Scanner::new(StrInput::new("key: value#bad\n")).collect::<Vec<Token<'_>>>();

		assert!(tokens.iter().any(\|Token(_, token)\| {
		matches!(token, TokenType::Scalar(ScalarStyle::Plain, value) if value == "value#bad")
		}));
		assert!(!tokens
		.iter()
		.any(\|Token(_, token)\| matches!(token, TokenType::Comment(_))));
		}

		#[test]
		fn parser_emits_full_line_indented_and_trailing_comment_events() {
		let yaml = "# top\n # indented\nkey: value # trailing\n#eof";
		let events = parser_events(yaml).expect("parser should accept comments");

		let comments: Vec<_> = events
		.iter()
		.filter_map(\|(event, span)\| match event {
		Event::Comment(text, _) => Some((text.as_ref(), span.slice(yaml))),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![
		(" top", Some("# top")),
		(" indented", Some("# indented")),
		(" trailing", Some("# trailing")),
		("eof", Some("#eof")),
		]
		);
		}

		#[test]
		fn parser_refines_comment_placements() {
		let yaml = "# above\na: b # right\n\n# free\n\nc: d\n...\n# last\n";
		let events = parser_events(yaml).expect("parser should accept comments");

		let comments: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Comment(text, placement) => Some((text.as_ref(), *placement)),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![
		(" above", Placement::Above),
		(" right", Placement::Right),
		(" free", Placement::Free),
		(" last", Placement::Last),
		]
		);
		}

		#[test]
		fn own_line_comment_before_invalid_token_is_emitted_before_error() {
		let mut parser = Parser::new_from_str("# c\n@\n");

		assert!(matches!(
		parser.next_event().unwrap().unwrap().0,
		Event::StreamStart
		));

		assert!(matches!(
		parser.next_event().unwrap().unwrap().0,
		Event::Comment(text, _) if text == " c"
		));

		let error = parser.next_event().unwrap().unwrap_err();
		assert!(error.info().contains("unexpected character"));
		}

		#[test]
		fn syntax_comment_before_invalid_token_is_emitted_before_error() {
		let mut parser = Parser::new_from_str("key: # c\n@\n");

		let comment = loop {
		let (event, _) = parser
		.next_event()
		.expect("parser should not end before comment")
		.expect("comment should be emitted before error");
		if let Event::Comment(text, _) = event {
		break text;
		}
		};

		assert_eq!(comment, " c");

		let error = parser.next_event().unwrap().unwrap_err();
		assert!(error.info().contains("unexpected character"));
		}

		#[test]
		fn parser_reports_comment_placements_in_nested_document() {
		let yaml = "\
		# root
		root:
		# child
		key: value # inline

		# detached

		next: value
		...
		# eof
		";
		let events = parser_events(yaml).expect("parser should accept comments");

		let comments: Vec<_> = events
		.iter()
		.filter_map(\|(event, span)\| match event {
		Event::Comment(text, placement) => Some((text.as_ref(), *placement, span.slice(yaml))),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![
		(" root", Placement::Above, Some("# root")),
		(" child", Placement::Above, Some("# child")),
		(" inline", Placement::Right, Some("# inline")),
		(" detached", Placement::Free, Some("# detached")),
		(" eof", Placement::Last, Some("# eof")),
		]
		);
		}

		#[test]
		fn parser_marks_consecutive_own_line_comments_as_above() {
		let yaml = "# first\n# second\nkey: value\n";
		let events = parser_events(yaml).expect("parser should accept comment block");

		let comments: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Comment(text, placement) => Some((text.as_ref(), *placement)),
		_ => None,
		})
		.collect();

		assert_eq!(
		comments,
		vec![(" first", Placement::Above), (" second", Placement::Above),]
		);
		}

		#[test]
		fn parser_emits_trailing_comment_after_plain_scalar_event() {
		let events = parser_events("key: value # trailing\n").expect("parser should emit events");

		let value_index = events
		.iter()
		.position(\|(event, _)\| {
		matches!(event, Event::Scalar(value, ScalarStyle::Plain, ..) if value == "value")
		})
		.expect("plain scalar event should be emitted");
		let comment_index = events
		.iter()
		.position(\|(event, _)\| matches!(event, Event::Comment(text, _) if text == " trailing"))
		.expect("comment event should be emitted");

		assert!(value_index < comment_index);
		}

		#[test]
		fn empty_mapping_value_after_comment_keeps_value_span() {
		let yaml = "key: # c\nnext: v\n";
		let events = parser_events(yaml).expect("parser should accept comments");

		let empty_value = first_empty_scalar_span(&events);
		let colon = yaml.find(':').unwrap();

		assert_eq!(empty_value.start.index(), colon);
		assert_eq!(empty_value.end.index(), colon);
		}

		#[test]
		fn empty_block_sequence_entry_after_comment_keeps_entry_span() {
		let yaml = "- # c\n- v\n";
		let events = parser_events(yaml).expect("parser should accept comments");

		let empty_item = first_empty_scalar_span(&events);
		let entry_marker = yaml.find("\n- v").unwrap();
		let second_dash = yaml.rfind('-').unwrap();

		assert_eq!(empty_item.start.index(), entry_marker);
		assert_eq!(empty_item.end.index(), entry_marker);
		assert_ne!(empty_item.start.index(), second_dash);
		}

		#[test]
		fn empty_indentless_sequence_entry_after_comment_keeps_entry_span() {
		let yaml = "key:\n- # c\n- v\n";
		let events = parser_events(yaml).expect("parser should accept comments");

		let empty_item = first_empty_scalar_span(&events);
		let entry_marker = yaml.find("\n- v").unwrap();
		let second_dash = yaml.rfind('-').unwrap();

		assert_eq!(empty_item.start.index(), entry_marker);
		assert_eq!(empty_item.end.index(), entry_marker);
		assert_ne!(empty_item.start.index(), second_dash);
		}

		#[test]
		fn empty_flow_mapping_value_after_comment_keeps_value_span() {
		let yaml = "{key: # c\n}";
		let events = parser_events(yaml).expect("parser should accept comments");

		let empty_value = first_empty_scalar_span(&events);
		let colon = yaml.find(':').unwrap();
		let closing_brace = yaml.rfind('}').unwrap();

		assert_eq!(empty_value.start.index(), colon);
		assert_eq!(empty_value.end.index(), colon);
		assert_ne!(empty_value.start.index(), closing_brace);
		}

		#[test]
		fn s3pd_comment_after_both_empty_preserves_span_order() {
		let yaml = "plain key: in-line value\n: # Both empty\n\"quoted key\":\n- entry\n";
		let events = parser_events(yaml).expect("S3PD should parse");

		assert_monotonic_spans(&events);
		}

		#[test]
		fn empty_block_sequence_entry_after_comment_preserves_span_order() {
		let yaml = "- # c\n- v\n";
		let events = parser_events(yaml).expect("sequence should parse");

		assert_monotonic_spans(&events);
		}

		#[test]
		fn empty_indentless_sequence_entry_after_comment_preserves_span_order() {
		let yaml = "key:\n- # c\n- v\n";
		let events = parser_events(yaml).expect("indentless sequence should parse");

		assert_monotonic_spans(&events);
		}

		#[test]
		fn empty_flow_mapping_value_after_comment_preserves_span_order() {
		let yaml = "{key: # c\n}";
		let events = parser_events(yaml).expect("flow mapping should parse");

		assert_monotonic_spans(&events);
		}

		#[test]
		fn later_empty_indentless_sequence_entry_after_comment_is_queued_before_next_key() {
		let yaml = "key:\n- first\n- # empty\nnext: value\n";
		let events = parser_events(yaml).expect("indentless sequence should parse");

		let names: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		Event::SequenceEnd => Some("SequenceEnd".into()),
		_ => None,
		})
		.collect();

		assert_eq!(
		names,
		vec![
		"Scalar(key)",
		"Scalar(first)",
		"Comment( empty)",
		"Scalar(~)",
		"SequenceEnd",
		"Scalar(next)",
		"Scalar(value)",
		]
		);
		}

		#[test]
		fn later_indentless_sequence_entry_after_comment_keeps_value_in_sequence() {
		let yaml = "key:\n- first\n- # value\n second\nnext: value\n";
		let events = parser_events(yaml).expect("indentless sequence should parse");

		let names: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		Event::SequenceEnd => Some("SequenceEnd".into()),
		_ => None,
		})
		.collect();

		assert_eq!(
		names,
		vec![
		"Scalar(key)",
		"Scalar(first)",
		"Comment( value)",
		"Scalar(second)",
		"SequenceEnd",
		"Scalar(next)",
		"Scalar(value)",
		]
		);
		}

		#[test]
		fn parser_handles_comments_in_flow_mapping_key_and_separator_states() {
		let yaml = "{? # key\n key\n: value, # comma\nnext: value}\n";
		let events = parser_events(yaml).expect("flow mapping should parse");

		let names: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		_ => None,
		})
		.collect();

		assert_eq!(
		names,
		vec![
		"Comment( key)",
		"Scalar(key)",
		"Scalar(value)",
		"Comment( comma)",
		"Scalar(next)",
		"Scalar(value)",
		]
		);
		}

		#[test]
		fn parser_handles_flow_mapping_value_after_comment() {
		let yaml = "{key: # value\n value}\n";
		let events = parser_events(yaml).expect("flow mapping should parse");

		let names: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		_ => None,
		})
		.collect();

		assert_eq!(
		names,
		vec!["Scalar(key)", "Comment( value)", "Scalar(value)"]
		);
		}

		#[test]
		fn parser_handles_implicit_flow_mapping_value_after_comment() {
		let yaml = "[key: # value\n value]\n";
		let events = parser_events(yaml).expect("implicit flow mapping should parse");

		let names: Vec<_> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::Scalar(value, ScalarStyle::Plain, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		_ => None,
		})
		.collect();

		assert_eq!(
		names,
		vec!["Scalar(key)", "Comment( value)", "Scalar(value)"]
		);
		}

		#[test]
		fn parser_preserves_empty_comment_payloads_and_crlf_span() {
		let yaml = "#\r\n# \n";
		let events = parser_events(yaml).expect("parser should accept empty comments");

		let comments: Vec<_> = events
		.iter()
		.filter_map(\|(event, span)\| match event {
		Event::Comment(text, _) => Some((text.as_ref(), span.slice(yaml))),
		_ => None,
		})
		.collect();

		assert_eq!(comments, vec![("", Some("#")), (" ", Some("# "))]);
		}

		#[test]
		fn parser_peek_returns_and_preserves_pending_comment_event() {
		let mut parser = Parser::new_from_str("# first\nkey: value\n");

		assert!(matches!(
		parser.next_event().unwrap().unwrap().0,
		Event::StreamStart
		));

		let first_peek = parser.peek().unwrap().unwrap().clone();
		let second_peek = parser.peek().unwrap().unwrap().clone();
		let next = parser.next_event().unwrap().unwrap();

		assert!(matches!(first_peek.0, Event::Comment(ref text, Placement::Above) if text == " first"));
		assert_eq!(first_peek, second_peek);
		assert_eq!(first_peek, next);
		}

		#[derive(Default)]
		struct CommentSink<'input> {
		comments: Vec<Cow<'input, str>>,
		}

		impl<'input> EventReceiver<'input> for CommentSink<'input> {
		fn on_event(&mut self, ev: Event<'input>) {
		if let Event::Comment(text, _) = ev {
		self.comments.push(text);
		}
		}
		}

		impl<'input> TryEventReceiver<'input> for CommentSink<'input> {
		type Error = ();

		fn on_event(&mut self, ev: Event<'input>) -> Result<(), Self::Error> {
		if let Event::Comment(text, _) = ev {
		self.comments.push(text);
		}
		Ok(())
		}
		}

		#[test]
		fn parser_load_and_try_load_deliver_comment_events() {
		let mut load_parser = Parser::new_from_str("# load\nkey: value\n");
		let mut load_sink = CommentSink::default();
		load_parser
		.load(&mut load_sink, true)
		.expect("load should deliver comments");

		let mut try_load_parser = Parser::new_from_str("# try\nkey: value\n");
		let mut try_load_sink = CommentSink::default();
		try_load_parser
		.try_load(&mut try_load_sink, true)
		.expect("try_load should deliver comments");

		assert_eq!(load_sink.comments, vec![Cow::Borrowed(" load")]);
		assert_eq!(try_load_sink.comments, vec![Cow::Borrowed(" try")]);
		}

		#[test]
		fn parser_emits_comments_around_markers_flow_collections_and_stream_end() {
		let yaml = "# before doc\n--- # after start\n[ # after flow start\n a, # after entry\n b\n] # after flow end\n... # after end\n# before stream end\n";
		let events = parser_events(yaml).expect("parser should emit comments in source order");

		let names: Vec<String> = events
		.iter()
		.filter_map(\|(event, _)\| match event {
		Event::StreamStart => Some("StreamStart".into()),
		Event::DocumentStart(_) => Some("DocumentStart".into()),
		Event::SequenceStart(..) => Some("SequenceStart".into()),
		Event::SequenceEnd => Some("SequenceEnd".into()),
		Event::DocumentEnd => Some("DocumentEnd".into()),
		Event::StreamEnd => Some("StreamEnd".into()),
		Event::Scalar(value, ..) => Some(format!("Scalar({value})")),
		Event::Comment(text, _) => Some(format!("Comment({text})")),
		Event::Nothing \| Event::Alias(_) \| Event::MappingStart(..) \| Event::MappingEnd => None,
		})
		.collect();

		assert_eq!(
		names,
		vec![
		"StreamStart",
		"Comment( before doc)",
		"DocumentStart",
		"Comment( after start)",
		"SequenceStart",
		"Comment( after flow start)",
		"Scalar(a)",
		"Comment( after entry)",
		"Scalar(b)",
		"SequenceEnd",
		"Comment( after flow end)",
		"DocumentEnd",
		"Comment( after end)",
		"Comment( before stream end)",
		"StreamEnd",
		]
		);
		}

		#[test]
		fn parser_keeps_comment_events_out_of_mapping_state_and_node_properties() {
		let yaml = "? # key\n: &a # anchor\n value\nref: *a # alias\n";
		let events =
		parser_events(yaml).expect("parser should preserve comments around mapping syntax");

		assert!(events
		.iter()
		.any(\|(event, _)\| matches!(event, Event::Comment(text, _) if text == " key")));
		assert!(events
		.iter()
		.any(\|(event, _)\| matches!(event, Event::Comment(text, _) if text == " anchor")));
		assert!(events
		.iter()
		.any(\|(event, _)\| matches!(event, Event::Comment(text, _) if text == " alias")));

		let anchored_value = events
		.iter()
		.find_map(\|(event, _)\| match event {
		Event::Scalar(value, _, anchor_id, _) if value == "value" => Some(*anchor_id),
		_ => None,
		})
		.expect("anchored scalar should be emitted");

		assert_ne!(anchored_value, 0);
		assert!(events
		.iter()
		.any(\|(event, _)\| matches!(event, Event::Alias(alias_id) if *alias_id == anchored_value)));
		}

+121

tests/coverage_edges.rs

		use granit_parser::{Event, Parser, ScalarStyle, ScanError, StructureStyle};

		fn parse_events(input: &str) -> Result<Vec<Event<'_>>, ScanError> {
		Parser::new_from_str(input)
		.map(\|event\| event.map(\|(event, _)\| event))
		.collect()
		}

		fn first_error_info(input: &str) -> String {
		for event in Parser::new_from_str(input) {
		if let Err(error) = event {
		return error.info().to_owned();
		}
		}
		panic!("expected parser error");
		}

		fn scalar_values(events: &[Event<'_>]) -> Vec<String> {
		events
		.iter()
		.filter_map(\|event\| {
		if let Event::Scalar(value, ..) = event {
		Some(value.to_string())
		} else {
		None
		}
		})
		.collect()
		}

		#[test]
		fn explicit_block_mapping_empty_key_is_null() {
		let events = parse_events("?\n: value\n").unwrap();

		assert_eq!(
		events,
		vec![
		Event::StreamStart,
		Event::DocumentStart(false),
		Event::MappingStart(StructureStyle::Block, 0, None),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::Scalar("value".into(), ScalarStyle::Plain, 0, None),
		Event::MappingEnd,
		Event::DocumentEnd,
		Event::StreamEnd,
		]
		);
		}

		#[test]
		fn flow_mapping_entry_without_colon_gets_null_value() {
		let events = parse_events("{foo}").unwrap();

		assert_eq!(scalar_values(&events), vec!["foo", "~"]);
		assert!(matches!(events.get(2), Some(Event::MappingStart(..))));
		assert!(events
		.iter()
		.any(\|event\| matches!(event, Event::MappingEnd)));
		}

		#[test]
		fn flow_sequence_explicit_mapping_can_omit_key_and_value() {
		let events = parse_events("[? ]").unwrap();

		assert_eq!(scalar_values(&events), vec!["~", "~"]);
		assert!(events
		.iter()
		.any(\|event\| matches!(event, Event::SequenceStart(..))));
		assert!(events
		.iter()
		.any(\|event\| matches!(event, Event::MappingStart(..))));
		}

		#[test]
		fn flow_sequence_explicit_mapping_can_omit_value() {
		let events = parse_events("[? foo]").unwrap();

		assert_eq!(scalar_values(&events), vec!["foo", "~"]);
		assert!(events
		.iter()
		.any(\|event\| matches!(event, Event::MappingEnd)));
		}

		#[test]
		fn flow_sequence_entry_requires_comma_before_next_collection() {
		assert_eq!(
		first_error_info("[a [b]]"),
		"while parsing a flow sequence, expected ',' or ']'"
		);
		}

		#[test]
		fn repeated_document_end_markers_do_not_start_empty_documents() {
		let events = parse_events("...\n...\n").unwrap();

		assert_eq!(events, vec![Event::StreamStart, Event::StreamEnd]);
		}

		#[test]
		fn block_mapping_rejects_unkeyed_content_after_nested_sequence() {
		assert_eq!(
		first_error_info("a:\n - b\n c\n"),
		"while parsing a block mapping, did not find expected key"
		);
		}

		#[test]
		fn flow_mapping_requires_comma_between_pairs() {
		assert_eq!(
		first_error_info("{a: b c: d}"),
		"while parsing a flow mapping, did not find expected ',' or '}'"
		);
		}

		#[test]
		fn block_sequence_rejects_explicit_mapping_entry_without_dash() {
		assert_eq!(
		first_error_info("- a\n? b\n"),
		"while parsing a block collection, did not find expected '-' indicator"
		);
		}

+1

-1

.cargo_vcs_info.json

		{
		"git": {
		"sha1": "25a68dd9632194904c086e18766c27a249768d87"
		"sha1": "c7df7015b8eae3aa1b7ab5abc44b3a3eb7a2524e"
		},
		"path_in_vcs": ""
		}

+2

-2

.github/workflows/api-compat.yml

		@@ -1,2 +0,2 @@
		name: API 0.0.1 compatibility
		name: API 0.0.3 compatibility

		@@ -14,3 +14,3 @@ on:
		env:
		BASELINE_REF: "0.0.1"
		BASELINE_REF: "0.0.3"

		@@ -17,0 +17,0 @@ jobs:

+2

-2

.github/workflows/ci.yml

		@@ -22,3 +22,3 @@ name: CI
		- name: Run clippy checks
		run: cargo clippy --all-targets -- -D warnings
		run: cargo clippy --all-targets --all-features -- -D warnings -D clippy::pedantic
		- name: Run format checks
		@@ -95,2 +95,2 @@ run: cargo fmt --check
		files: lcov.info
		fail_ci_if_error: true
		fail_ci_if_error: true

+1

-1

.github/workflows/release.yml

		@@ -65,3 +65,3 @@ name: Release
		- name: Run clippy
		run: cargo clippy --offline --all-targets --all-features -- -D warnings
		run: cargo clippy --offline --all-targets --all-features -- -D warnings -D clippy::pedantic

		@@ -68,0 +68,0 @@ - name: Run tests

+1

-1

Cargo.lock

		@@ -115,3 +115,3 @@ # This file is automatically @generated by Cargo.
		name = "granit-parser"
		version = "0.0.2"
		version = "0.0.3"
		dependencies = [
		@@ -118,0 +118,0 @@ "arraydeque",

+18

-2

Cargo.toml

		@@ -16,3 +16,3 @@ # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
		name = "granit-parser"
		version = "0.0.2"
		version = "0.0.3"
		authors = [
		@@ -34,3 +34,3 @@ "Ethiraric <ethiraric@gmail.com>",
		autobenches = false
		description = "A YAML parser in pure Rust with strict compliance"
		description = "A YAML parser in pure Rust with comment and style support"
		documentation = "https://docs.rs/granit-parser/latest/granit_parser"
		@@ -69,2 +69,10 @@ readme = "README.md"

		[[example]]
		name = "custom_tags"
		path = "examples/custom_tags.rs"

		[[example]]
		name = "include_stack"
		path = "examples/include_stack.rs"

		[[test]]
		@@ -87,2 +95,6 @@ name = "basic"
		[[test]]
		name = "comment"
		path = "tests/comment.rs"

		[[test]]
		name = "comment_intercepts_line"
		@@ -92,2 +104,6 @@ path = "tests/comment_intercepts_line.rs"
		[[test]]
		name = "coverage_edges"
		path = "tests/coverage_edges.rs"

		[[test]]
		name = "document_markers"
		@@ -94,0 +110,0 @@ path = "tests/document_markers.rs"

+32

-0

CHANGELOG.md

		# Changelog

		## v0.0.3

		- Added `Span::slice(&str)` for convenient source extraction from parser-emitted
		spans when byte offsets are available.
		- Added small `Tag` inspection helpers (`is_yaml_core_schema_tag`, `is_custom`,
		`parts`) for YAML-core-namespace and non-core-namespace tag consumers.
		- Added `Event` inspection helpers (`anchor_id`, `alias_id`, `tag`, `scalar`,
		`is_node`) for ergonomic access to per-event node metadata. `anchor_id`
		returns `Option<usize>` for events that define an anchor; `alias_id` returns
		`Option<usize>` for aliases that reference an anchor; `is_node` returns `true`
		for any event that produces a value in the document tree.
		- Added comment events. Comments are emitted as parser
		`Event::Comment(text, placement)` events with normal companion spans, and as
		scanner comment tokens carrying `Placement` metadata. This is an event-stream
		change: consumers building YAML data trees should ignore comment events, while
		formatting/linting tools can use the placement hint and comment span.
		- Added `StructureStyle` metadata for sequences and mappings, exposed on
		`Event::SequenceStart(style, anchor_id, tag)` and
		`Event::MappingStart(style, anchor_id, tag)` events.
		- Added `ParserStack::push_include` as an include-oriented alias for
		`ParserStack::resolve`.
		- Added a custom-tag example showing how to inspect application tags such as
		`!degrees` from the event stream.
		- Added an include-stack example demonstrating `!include`-style resolution
		through `ParserStack`.
		- Updated the README minimal example to show `Span::slice` alongside byte
		ranges.
		- Fixed the span of quoted (single- and double-quoted) scalars to end at the
		closing quote rather than after trailing whitespace and an optional
		end-of-line comment. This makes `Span::slice` return only the scalar source.
		- Fixed pedantic Clippy warnings and switched into pedantic mode.

		## v0.0.2
		@@ -4,0 +36,0 @@

+84

-49

README.md

		@@ -12,3 +12,3 @@ # granit-parser
		[![crates.io](https://img.shields.io/crates/v/granit-parser.svg)](https://crates.io/crates/granit-parser)
		[![0.0.1 compatible (see API note)](https://github.com/bourumir-wyngs/granit-parser/actions/workflows/api-compat.yml/badge.svg)](https://github.com/bourumir-wyngs/granit-parser/actions/workflows/api-compat.yml)
		[![0.0.3 compatible (see API note)](https://github.com/bourumir-wyngs/granit-parser/actions/workflows/api-compat.yml/badge.svg)](https://github.com/bourumir-wyngs/granit-parser/actions/workflows/api-compat.yml)
		[![crates.io](https://img.shields.io/crates/d/granit-parser.svg)](https://crates.io/crates/granit-parser)
		@@ -20,3 +20,3 @@

		granit-parser is both YAML 1.1 and 1.2 compliant parser in pure Rust with strict compliance, no-std support, and spans for parser events.
		granit-parser is both YAML 1.1 and 1.2 compliant parser in pure Rust with strict compliance, comment and style support, no-std support, and spans for parser events. “Granit” is a correct word in many European languages (English granite).

		@@ -27,7 +27,8 @@ This crate started as a fork of [saphyr-parser](https://crates.io/crates/saphyr-parser) that descends from [yaml-rust](https://github.com/chyh1990/yaml-rust), with influences from [libyaml](https://crates.io/crates/libyaml) and [yaml-cpp](https://github.com/jbeder/yaml-cpp). The project has since diverged significantly and is now maintained as an independent project.

		* full compliance with the [yaml-test-suite](https://github.com/yaml/yaml-test-suite), including correctness in edge cases
		* [`Comment`](https://docs.rs/granit-parser/latest/granit_parser/struct.Comment.html) support and [`StructureStyle`](https://docs.rs/granit-parser/latest/granit_parser/enum.StructureStyle.html) information. This is for linting, formatting, and analysis.
		* compliance with the [yaml-test-suite](https://github.com/yaml/yaml-test-suite), including correctness in edge cases
		* compatibility with real-world YAML usage
		* quickly incorporate the changes we need for the upstream dependency [serde-saphyr](https://crates.io/crates/serde-saphyr).

		`granit-parser`’s public API is very similar to that of `saphyr-parser`, so it is typically an easy replacement. However, some changes are still breaking (crate rename, MSRV bump, lifetimes on events, Cow payloads, etc.).
		`granit-parser`’s 0.0.1 or 0.0.2 public API is very similar to that of `saphyr-parser`, so it is typically an easy replacement. Later versions emit style and comment information, you need to adjust your code to handle or discard them.

		@@ -38,6 +39,8 @@ See [releases](https://github.com/bourumir-wyngs/granit-parser/releases)

		`Parser::new_from_str` returns an iterator of `(Event, Span)` pairs. If you only care about parser events, you can ignore the span and match on the emitted `Event` values:
		[`Parser::new_from_str`](https://docs.rs/granit-parser/latest/granit_parser/struct.Parser.html#method.new_from_str) returns an iterator of ([`Event`](https://docs.rs/granit-parser/latest/granit_parser/enum.Event.html), [`Span`](https://docs.rs/granit-parser/latest/granit_parser/struct.Span.html)) pairs. The event helpers expose common node metadata, and spans provide byte ranges plus source slices:

		Comments are emitted as `Event::Comment(text, placement)`. They are presentation metadata for tools such as linters and formatters, not YAML data nodes, so consumers that build YAML values should filter them out. The companion `Span` for a comment covers the whole source comment, including `#` and excluding the line break; when parsing from `Parser::new_from_str`, `span.slice(yaml)` returns that source comment text.

		```rust
		use granit_parser::{Event, Parser};
		use granit_parser::Parser;

		@@ -52,19 +55,26 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
		[40.7128, -74.0060]: remote
		music: "\uD834\uDD1E\uD83C\uDFB5\uD83C\uDFB6" # JSON-style \uXXXX surrogate pairs

		# JSON-style \uXXXX surrogate pairs:
		music: "\uD834\uDD1E\uD83C\uDFB5\uD83C\uDFB6"
		"#;

		for next in Parser::new_from_str(yaml) {
		let (event, _span) = next?;
		let (event, span) = next?;

		match &event {
		Event::SequenceStart(_, Some(tag)) => {
		println!("sequence tag: {}{}", tag.handle, tag.suffix);
		if let Some(tag) = event.tag() {
		if let Some((value, _style)) = event.scalar() {
		println!(
		"scalar tag: {tag} core-str={} for {value:?}",
		tag.is_yaml_core_schema_tag("str")
		);
		} else if event.is_node() {
		println!("node tag: {tag} custom={}", tag.is_custom());
		}
		Event::Scalar(value, _, _, Some(tag)) => {
		println!("scalar tag: {}{} for {value:?}", tag.handle, tag.suffix);
		}
		_ => {}
		}

		println!("{event:?}");
		println!(
		"{event:?} bytes={:?} source={:?}",
		span.byte_range(),
		span.slice(yaml)
		);
		}
		@@ -79,30 +89,32 @@
		```text
		StreamStart
		DocumentStart(false)
		MappingStart(0, None)
		Scalar("items", Plain, 0, None)
		sequence tag: !shopping
		SequenceStart(0, Some(Tag { handle: "!", suffix: "shopping" }))
		Scalar("milk", Plain, 0, None)
		scalar tag: tag:yaml.org,2002:str for "bread"
		Scalar("bread", Plain, 0, Some(Tag { handle: "tag:yaml.org,2002:", suffix: "str" }))
		SequenceEnd
		Scalar("locations", Plain, 0, None)
		MappingStart(0, None)
		SequenceStart(0, None)
		Scalar("47.3769", Plain, 0, None)
		Scalar("8.5417", Plain, 0, None)
		SequenceEnd
		Scalar("local", Plain, 0, None)
		SequenceStart(0, None)
		Scalar("40.7128", Plain, 0, None)
		Scalar("-74.0060", Plain, 0, None)
		SequenceEnd
		Scalar("remote", Plain, 0, None)
		MappingEnd
		Scalar("music", Plain, 0, None)
		Scalar("𝄞🎵🎶", DoubleQuoted, 0, None)
		MappingEnd
		DocumentEnd
		StreamEnd
		StreamStart bytes=Some(0..0) source=Some("")
		DocumentStart(false) bytes=Some(1..1) source=Some("")
		MappingStart(Block, 0, None) bytes=Some(1..1) source=Some("")
		Scalar("items", Plain, 0, None) bytes=Some(1..6) source=Some("items")
		node tag: !shopping custom=true
		SequenceStart(Block, 0, Some(Tag { handle: "!", suffix: "shopping" })) bytes=Some(20..20) source=Some("")
		Scalar("milk", Plain, 0, None) bytes=Some(22..26) source=Some("milk")
		scalar tag: tag:yaml.org,2002:str core-str=true for "bread"
		Scalar("bread", Plain, 0, Some(Tag { handle: "tag:yaml.org,2002:", suffix: "str" })) bytes=Some(37..42) source=Some("bread")
		SequenceEnd bytes=Some(43..43) source=Some("")
		Scalar("locations", Plain, 0, None) bytes=Some(43..52) source=Some("locations")
		Comment(" Example with composite keys", Right) bytes=Some(54..83) source=Some("# Example with composite keys")
		MappingStart(Block, 0, None) bytes=Some(86..86) source=Some("")
		SequenceStart(Flow, 0, None) bytes=Some(86..87) source=Some("[")
		Scalar("47.3769", Plain, 0, None) bytes=Some(87..94) source=Some("47.3769")
		Scalar("8.5417", Plain, 0, None) bytes=Some(96..102) source=Some("8.5417")
		SequenceEnd bytes=Some(102..103) source=Some("]")
		Scalar("local", Plain, 0, None) bytes=Some(105..110) source=Some("local")
		SequenceStart(Flow, 0, None) bytes=Some(113..114) source=Some("[")
		Scalar("40.7128", Plain, 0, None) bytes=Some(114..121) source=Some("40.7128")
		Scalar("-74.0060", Plain, 0, None) bytes=Some(123..131) source=Some("-74.0060")
		SequenceEnd bytes=Some(131..132) source=Some("]")
		Scalar("remote", Plain, 0, None) bytes=Some(134..140) source=Some("remote")
		Comment(" JSON-style \\uXXXX surrogate pairs:", Above) bytes=Some(142..178) source=Some("# JSON-style \\uXXXX surrogate pairs:")
		MappingEnd bytes=Some(179..179) source=Some("")
		Scalar("music", Plain, 0, None) bytes=Some(179..184) source=Some("music")
		Scalar("𝄞🎵🎶", DoubleQuoted, 0, None) bytes=Some(186..224) source=Some("\"\\uD834\\uDD1E\\uD83C\\uDFB5\\uD83C\\uDFB6\"")
		MappingEnd bytes=Some(225..225) source=Some("")
		DocumentEnd bytes=Some(225..225) source=Some("")
		StreamEnd bytes=Some(225..225) source=Some("")
		```
		@@ -112,8 +124,25 @@

		Use [`try_load`](https://docs.rs/granit-parser/latest/granit_parser/struct.Parser.html#method.try_load)
		when a receiver may return a validation or application error and parsing should
		stop immediately. It accepts
		[`TryEventReceiver`](https://docs.rs/granit-parser/latest/granit_parser/trait.TryEventReceiver.html)
		or
		[`TrySpannedEventReceiver`](https://docs.rs/granit-parser/latest/granit_parser/trait.TrySpannedEventReceiver.html)
		and returns
		[`TryLoadError`](https://docs.rs/granit-parser/latest/granit_parser/enum.TryLoadError.html) to
		distinguish parser errors from receiver errors.

		Event-only receivers receive comment events as `Event::Comment(text, placement)`.
		Spanned receivers receive the same event plus the comment span in
		[`on_event`](https://docs.rs/granit-parser/latest/granit_parser/trait.SpannedEventReceiver.html#tymethod.on_event).
		When using [`resolve`](https://docs.rs/granit-parser/latest/granit_parser/parser_stack/struct.ParserStack.html#method.resolve)
		or [`push_include`](https://docs.rs/granit-parser/latest/granit_parser/parser_stack/struct.ParserStack.html#method.push_include)
		on `ParserStack`, comment events
		from included documents are forwarded through the normal event stream. Their
		spans remain local to the included source, matching the existing span behavior
		for other included-document events.

		Use the iterator API when the caller should pull events and decide when to stop
		parsing. Use `Parser::load` for infallible receiver-style event sinks. If a
		receiver may return a validation or application error and parsing should stop
		immediately, use `Parser::try_load` with `TryEventReceiver` or
		`TrySpannedEventReceiver`; it returns `TryLoadError::Scan` for parser errors and
		`TryLoadError::Receiver` for receiver errors.
		parsing. [`load`](https://docs.rs/granit-parser/latest/granit_parser/struct.Parser.html#method.load)
		is `infallible`.

		@@ -205,2 +234,8 @@ ## Key differences from saphyr-parser

		### Improved ergonomics
		Release 0.0.3 includes ergonomic helpers such as `Event::tag`, `Event::scalar`,
		`Event::anchor_id`, `Event::alias_id`, `Event::is_node`, `Tag::parts`,
		`Tag::is_custom`, `Tag::is_yaml_core_schema_tag`, `Span::slice`, and
		`ParserStack::push_include`. See CHANGELOG.md for details.

		## Tools
		@@ -207,0 +242,0 @@

+27

-3

src/char_traits.rs

		@@ -31,3 +31,3 @@ //! Holds functions to determine if a character belongs to a specific character set.

		/// Check whether the character is nil, a linebreak or a whitespace.
		/// Check whether the character is nil, a line break, or whitespace.
		///
		@@ -41,3 +41,3 @@ /// `\0`, ` `, `\t`, `\n`, `\r`

		/// Check whether the character is an ascii digit.
		/// Check whether the character is an ASCII digit.
		#[inline]
		@@ -53,3 +53,3 @@ #[must_use]
		/// Note: This is slightly more permissive than YAML's `ns-word-char` (which excludes `_`).
		/// For strict `ns-word-char` compliance, use [`is_word_char`] instead.
		/// For strict `ns-word-char` compliance, use `is_word_char` instead.
		///
		@@ -158,1 +158,25 @@ /// Matches: `[0-9a-zA-Z_-]`
		}

		#[cfg(test)]
		mod tests {
		use super::*;

		#[test]
		fn printable_ranges_include_private_and_supplementary_planes() {
		assert!(is_printable('\u{E000}'));
		assert!(is_printable('\u{10FFFF}'));
		assert!(is_yaml_non_break('\u{10000}'));
		assert!(!is_yaml_non_break('\u{FEFF}'));
		assert!(!is_yaml_non_break('\n'));
		}

		#[test]
		fn word_uri_and_tag_character_sets_are_distinct() {
		assert!(is_word_char('-'));
		assert!(!is_word_char('_'));
		assert!(is_uri_char('_'));
		assert!(is_uri_char('%'));
		assert!(!is_tag_char('!'));
		assert!(!is_tag_char('['));
		}
		}

+34

-20

src/input.rs

		@@ -42,8 +42,6 @@ //! Utilities to create a source of input to the parser.
		///
		/// Hiding the input's implementation behind this trait allows mostly:
		/// * For input-specific optimizations (for instance, using `str` methods instead of manually
		/// transferring one `char` at a time to a buffer).
		/// * To return `&str`s referencing the input string, thus avoiding potentially costly
		/// allocations. Should users need an owned version of the data, they can always `.to_owned()`
		/// their YAML object.
		/// Hiding the input's implementation behind this trait allows input-specific optimizations, such
		/// as using `str` methods instead of manually transferring one `char` at a time to a buffer.
		/// Implementations with stable backing storage can also return borrowed `&str` slices and avoid
		/// allocating token values.
		pub trait Input {
		@@ -65,7 +63,7 @@ /// A hint to the input source that we will need to read `count` characters.

		/// Return the capacity of the buffer in `self`.
		/// Return the maximum number of characters this input can buffer for lookahead.
		#[must_use]
		fn bufmaxlen(&self) -> usize;

		/// Return whether the buffer (!= stream) is empty.
		/// Return whether the lookahead buffer is empty.
		#[inline]
		@@ -83,3 +81,3 @@ #[must_use]

		/// Read a non-breakz a character from the input stream and return it directly.
		/// Read a non-breakz character from the input stream and return it directly.
		///
		@@ -96,3 +94,3 @@ /// The internal buffer (if any) is bypassed.

		/// Consume the next `count` character.
		/// Consume the next `count` characters.
		fn skip_n(&mut self, count: usize);
		@@ -114,3 +112,3 @@
		///
		/// This function assumes that the n-th character in the input has already been fetched through
		/// This function assumes that the `n`-th character in the input has already been fetched through
		/// [`Input::lookahead`].
		@@ -191,3 +189,3 @@ #[must_use]
		///
		/// This function assumes that the n-th character in the input has already been fetched through
		/// This function assumes that the `n`-th character in the input has already been fetched through
		/// [`Input::lookahead`].
		@@ -224,3 +222,3 @@ #[inline]
		///
		/// This function assumes that the next 4 characters in the input has already been fetched
		/// This function assumes that the next 4 characters in the input have already been fetched
		/// through [`Input::lookahead`].
		@@ -237,3 +235,3 @@ #[inline]
		///
		/// This function assumes that the next 4 characters in the input has already been fetched
		/// This function assumes that the next 4 characters in the input have already been fetched
		/// through [`Input::lookahead`].
		@@ -249,3 +247,3 @@ #[inline]
		///
		/// This function assumes that the next 4 characters in the input has already been fetched
		/// This function assumes that the next 4 characters in the input have already been fetched
		/// through [`Input::lookahead`].
		@@ -259,4 +257,6 @@ #[inline]

		/// Skip yaml whitespace at most up to eol. Also skips comments. Advances the input.
		/// Skip YAML whitespace up to the end of the current line.
		///
		/// Inline comments are consumed only after at least one preceding YAML whitespace character.
		///
		/// # Return
		@@ -266,3 +266,3 @@ /// Return a tuple with the number of characters that were consumed and the result of skipping
		/// since no end-of-line character will be consumed.
		/// See [`SkipTabs`] For more details on the success variant.
		/// See [`SkipTabs`] for more details on the success variant.
		///
		@@ -559,3 +559,3 @@ /// # Errors
		///
		/// Although tab is a valid yaml whitespace, it doesn't always behave the same as a space.
		/// Although tab is valid YAML whitespace, it does not always behave the same as a space.
		#[derive(Copy, Clone, Eq, PartialEq)]
		@@ -571,3 +571,3 @@ pub enum SkipTabs {
		bool,
		/// Whether at least 1 valid yaml whitespace has been encountered.
		/// Whether at least one valid YAML whitespace character has been encountered.
		bool,
		@@ -597,3 +597,3 @@ ),
		mod tests {
		use super::Input;
		use super::{Input, SkipTabs};

		@@ -650,2 +650,16 @@ struct MinimalInput;
		}

		#[test]
		fn default_skip_ws_to_eol_rejects_unseparated_comment() {
		let mut input = super::buffered::BufferedInput::new("#comment\n".chars());

		let (consumed, result) = input.skip_ws_to_eol(SkipTabs::Yes);

		assert_eq!(consumed, 0);
		assert_eq!(
		result.err(),
		Some("comments must be separated from other tokens by whitespace")
		);
		assert_eq!(input.peek(), '#');
		}
		}

+4

-4

src/input/buffered.rs

		@@ -23,7 +23,7 @@ use crate::char_traits::is_breakz;
		/// characters at a time and sometimes pushing some back into the stream.
		/// There is no "easy" way of doing this without itertools. In order to avoid pulling the entierty
		/// of itertools for one method, we use this structure.
		/// Doing this directly with iterator adapters would require pulling in all of `itertools` for one
		/// method, so this structure keeps the buffering local.
		#[allow(clippy::module_name_repetitions)]
		pub struct BufferedInput<T: Iterator<Item = char>> {
		/// The iterator source,
		/// The iterator source.
		input: T,
		@@ -35,3 +35,3 @@ /// Buffer for the next characters to consume.
		impl<T: Iterator<Item = char>> BufferedInput<T> {
		/// Create a new [`BufferedInput`] with the given input.
		/// Create a new [`BufferedInput`] over the given character iterator.
		pub fn new(input: T) -> Self {
		@@ -38,0 +38,0 @@ Self {

+31

-18

src/input/str.rs

		@@ -7,3 +7,3 @@ use crate::{

		/// A parser input that uses a `&str` as source.
		/// A parser input backed by a `&str`.
		#[allow(clippy::module_name_repetitions)]
		@@ -16,3 +16,3 @@ pub struct StrInput<'a> {
		original: &'a str,
		/// The input str buffer.
		/// The remaining input slice.
		///
		@@ -24,4 +24,4 @@ /// This is a moving window into [`Self::original`]. All consuming operations advance this
		///
		/// We must however keep track of how many characters the parser asked us to look ahead for so
		/// that we can return the correct value in [`Self::buflen`].
		/// This tracks how many characters the parser asked us to look ahead for so we can return the
		/// correct value in [`Self::buflen`].
		lookahead: usize,
		@@ -31,3 +31,3 @@ }
		impl<'a> StrInput<'a> {
		/// Create a new [`StrInput`] with the given str.
		/// Create a new [`StrInput`] over the given string slice.
		#[must_use]
		@@ -57,4 +57,4 @@ pub fn new(input: &'a str) -> Self {
		// We already have all characters that we need.
		// We cannot add '\0's to the buffer should we prematurely reach EOF.
		// Returning '\0's befalls the character-retrieving functions.
		// We cannot add '\0's to the buffer when we reach EOF.
		// Character-retrieving functions return '\0' when they read past EOF.
		self.lookahead = self.lookahead.max(x);
		@@ -203,3 +203,3 @@ }
		} else {
		// Since all characters we look for are ascii, we can directly use the byte API of str.
		// Since all characters we look for are ASCII, we can directly use the byte API of str.
		let bytes = self.buffer.as_bytes();
		@@ -218,3 +218,3 @@ (bytes.len() == 3 \|\| matches!(bytes[3], b' ' \| b'\t' \| 0 \| b'\n' \| b'\r'))
		} else {
		// Since all characters we look for are ascii, we can directly use the byte API of str.
		// Since all characters we look for are ASCII, we can directly use the byte API of str.
		let bytes = self.buffer.as_bytes();
		@@ -233,3 +233,3 @@ (bytes.len() == 3 \|\| matches!(bytes[3], b' ' \| b'\t' \| 0 \| b'\n' \| b'\r'))
		} else {
		// Since all characters we look for are ascii, we can directly use the byte API of str.
		// Since all characters we look for are ASCII, we can directly use the byte API of str.
		let bytes = self.buffer.as_bytes();
		@@ -250,4 +250,3 @@ (bytes.len() == 3 \|\| matches!(bytes[3], b' ' \| b'\t' \| 0 \| b'\n' \| b'\r'))

		// This ugly pair of loops is the fastest way of trimming spaces (and maybe tabs) I found
		// while keeping track of whether we encountered spaces and/or tabs.
		// Separate loops keep the fast space-only path while still tracking whether tabs were seen.
		if skip_tabs == SkipTabs::Yes {
		@@ -272,3 +271,3 @@ loop {

		// All characters consumed were ascii. We can use the byte length difference to count the
		// All characters consumed were ASCII. We can use the byte length difference to count the
		// number of whitespace ignored.
		@@ -523,5 +522,5 @@ let mut chars_consumed = self.buffer.len() - new_str.len();
		///
		/// This does not correspond to any allocated buffer size. In practice, the scanner can withdraw
		/// any character they want. If it's within the input buffer, the given character is returned,
		/// otherwise `\0` is returned.
		/// This does not correspond to any allocated buffer size. In practice, the scanner may request any
		/// character in the virtual buffer: characters inside the input are returned as-is, and positions
		/// past EOF return `\0`.
		///
		@@ -535,4 +534,4 @@ /// The number of characters we are asked to retrieve in [`lookahead`] depends on the buffer size
		///
		/// This create a complex situation where [`bufmaxlen`] influences what value [`lookahead`] is
		/// called with, which in turns dictates what [`buflen`] returns. In order to avoid breaking any
		/// This creates a complex situation where [`bufmaxlen`] influences what value [`lookahead`] is
		/// called with, which in turn dictates what [`buflen`] returns. In order to avoid breaking any
		/// function, we return this constant in [`bufmaxlen`] which, since the input is processed one line
		@@ -672,2 +671,16 @@ /// at a time, should fit what we expect to be a good balance between memory consumption and what
		#[test]
		fn skip_ws_to_eol_rejects_unseparated_comment() {
		let mut input = StrInput::new("# comment\n");

		let (consumed, result) = input.skip_ws_to_eol(SkipTabs::Yes);

		assert_eq!(consumed, 0);
		assert_eq!(
		result.err(),
		Some("comments must be separated from other tokens by whitespace")
		);
		assert_eq!(input.peek(), '#');
		}

		#[test]
		fn fetch_while_is_alpha_is_ascii_only() {
		@@ -674,0 +687,0 @@ let mut input = StrInput::new("abc_123-é");

+33

-8

src/lib.rs

		@@ -9,2 +9,4 @@ // Copyright 2015, Yuheng Chen.
		//! [`Event`] values paired with their source [`Span`].
		//! Comments are emitted as [`Event::Comment`]. They are presentation metadata, not YAML data
		//! nodes, so consumers building YAML value trees should ignore them.
		//!
		@@ -20,13 +22,30 @@ //! Add it to your project:
		//! ```rust
		//! use granit_parser::{Event, Parser};
		//! use granit_parser::{Event, Parser, Placement};
		//!
		//! # fn main() -> Result<(), granit_parser::ScanError> {
		//! let yaml = "items:\n - milk\n - bread\n";
		//! let yaml = r#"# header
		//! items: # inline
		//! - milk
		//! - bread
		//! "#;
		//! let mut comments = Vec::new();
		//!
		//! for next in Parser::new_from_str(yaml) {
		//! let (event, _span) = next?;
		//! if let Event::Scalar(value, ..) = event {
		//! println!("{value}");
		//! let (event, span) = next?;
		//! if let Event::Comment(text, placement) = event {
		//! comments.push((
		//! text.into_owned(),
		//! placement,
		//! span.slice(yaml).unwrap().to_owned(),
		//! ));
		//! }
		//! }
		//!
		//! assert_eq!(
		//! comments,
		//! [
		//! (" header".to_owned(), Placement::Above, "# header".to_owned()),
		//! (" inline".to_owned(), Placement::Right, "# inline".to_owned()),
		//! ]
		//! );
		//! # Ok(())
		@@ -36,2 +55,6 @@ //! # }
		//!
		//! For comment events, the companion [`Span`] covers the whole source comment, including `#` and
		//! excluding the line break. With [`Parser::new_from_str`], [`Span::slice`] returns that source
		//! comment text.
		//!
		//! # Features
		@@ -71,5 +94,7 @@ //! Note: This crate's MSRV is `1.81.0`.
		pub use crate::parser::{
		Event, EventReceiver, Parser, ParserTrait, SpannedEventReceiver, Tag, TryEventReceiver,
		TryLoadError, TrySpannedEventReceiver,
		Event, EventReceiver, Parser, ParserTrait, SpannedEventReceiver, StructureStyle, Tag,
		TryEventReceiver, TryLoadError, TrySpannedEventReceiver,
		};
		pub use crate::scanner::{Marker, ScalarStyle, ScanError, Scanner, Span, Token, TokenType};
		pub use crate::scanner::{
		Comment, Marker, Placement, ScalarStyle, ScanError, Scanner, Span, Token, TokenType,
		};

+70

-31

src/parser_stack.rs

		@@ -17,3 +17,3 @@ use crate::{
		impl<'input> ReplayParser<'input> {
		/// Creates a new `ReplayParser`.
		/// Create a parser that replays `events` and starts anchor allocation at `anchor_offset`.
		#[must_use]
		@@ -29,3 +29,3 @@ pub fn new(events: Vec<(Event<'input>, Span)>, anchor_offset: usize) -> Self {

		/// Get the current anchor offset count.
		/// Return the next anchor ID that should be assigned after replayed events.
		#[must_use]
		@@ -36,3 +36,3 @@ pub fn get_anchor_offset(&self) -> usize {

		/// Set the current anchor offset count.
		/// Set the next anchor ID that should be assigned after replayed events.
		pub fn set_anchor_offset(&mut self, offset: usize) {
		@@ -45,4 +45,4 @@ self.anchor_offset = offset;
		Event::Scalar(_, _, anchor_id, _)
		\| Event::SequenceStart(anchor_id, _)
		\| Event::MappingStart(anchor_id, _) => *anchor_id,
		\| Event::SequenceStart(_, anchor_id, _)
		\| Event::MappingStart(_, anchor_id, _) => *anchor_id,
		_ => 0,
		@@ -104,14 +104,14 @@ };
		{
		/// A parser over a string input.
		/// A parser over borrowed string input.
		String {
		/// The parser itself.
		/// Parser currently producing events for this stack entry.
		parser: Parser<'input, StrInput<'input>>,
		/// The name of the parser.
		/// Human-readable source name returned by [`ParserStack::stack`].
		name: String,
		},
		/// A parser over an iterator input.
		/// A parser over an iterator of characters.
		Iter {
		/// The parser itself.
		/// Parser currently producing events for this stack entry.
		parser: Parser<'static, BufferedInput<I>>,
		/// The name of the parser.
		/// Human-readable source name returned by [`ParserStack::stack`].
		name: String,
		@@ -121,5 +121,5 @@ },
		Custom {
		/// The parser itself.
		/// Parser currently producing events for this stack entry.
		parser: Parser<'input, T>,
		/// The name of the parser.
		/// Human-readable source name returned by [`ParserStack::stack`].
		name: String,
		@@ -129,5 +129,5 @@ },
		Replay {
		/// The replay parser itself.
		/// Replay parser currently producing pre-collected events for this stack entry.
		parser: ReplayParser<'input>,
		/// The name of the parser.
		/// Human-readable source name returned by [`ParserStack::stack`].
		name: String,
		@@ -161,3 +161,3 @@ },

		/// A parser implementation that utilizes a stack for parsing.
		/// A parser implementation that uses a stack for include-style parsing.
		///
		@@ -170,2 +170,7 @@ /// Note: `ParserStack` deliberately suppresses nested [`Event::StreamStart`] /
		/// That is exactly what we want for `!include`-style subtree injection.
		///
		/// Included parser events, including [`Event::Comment`] events, are replayed through the same
		/// event stream as parent events. Their [`Span`] values remain local to the included source, just
		/// like every other event span from an included parser. `ParserStack` does not attach file names,
		/// source IDs, or other include provenance to events or spans.
		pub struct ParserStack<'input, I = core::iter::Empty<char>, T = StrInput<'input>>
		@@ -199,3 +204,5 @@ where

		/// Sets the include resolver for this stack.
		/// Set the resolver used by [`Self::resolve`] and [`Self::push_include`].
		///
		/// The resolver receives the include name and returns the included YAML source text.
		pub fn set_resolver(
		@@ -210,2 +217,6 @@ &mut self,
		///
		/// Comment events from the included content are preserved. Their spans are local to the
		/// included content returned by the resolver, matching the existing behavior for all included
		/// document events.
		///
		/// # Errors
		@@ -239,3 +250,19 @@ /// Returns `ScanError` if no resolver is configured, include resolution fails, or the

		/// Pushes a string parser onto the stack.
		/// Resolves an include by name and pushes the resulting parser onto the stack.
		///
		/// This is an alias for [`Self::resolve`] with a name that reads naturally in
		/// include-oriented consumers: `stack.push_include("config.yaml")?`.
		/// Comment spans from the included content are local to that included source.
		///
		/// # Errors
		/// Returns `ScanError` if no resolver is configured, include resolution fails, or the
		/// included content cannot be parsed.
		pub fn push_include(&mut self, include_name: &str) -> Result<(), ScanError> {
		self.resolve(include_name)
		}

		/// Push a string parser onto the stack.
		///
		/// The pushed parser inherits the current anchor offset so anchors remain unique across stacked
		/// sources. `name` is returned by [`Self::stack`] for diagnostics.
		pub fn push_str_parser(&mut self, mut parser: Parser<'input, StrInput<'input>>, name: String) {
		@@ -248,3 +275,6 @@ if let Some(parent) = self.parsers.last() {

		/// Pushes an iterator parser onto the stack.
		/// Push an iterator-backed parser onto the stack.
		///
		/// The pushed parser inherits the current anchor offset so anchors remain unique across stacked
		/// sources. `name` is returned by [`Self::stack`] for diagnostics.
		pub fn push_iter_parser(
		@@ -261,3 +291,6 @@ &mut self,

		/// Pushes a custom parser onto the stack.
		/// Push a custom-input parser onto the stack.
		///
		/// The pushed parser inherits the current anchor offset so anchors remain unique across stacked
		/// sources. `name` is returned by [`Self::stack`] for diagnostics.
		pub fn push_custom_parser(&mut self, mut parser: Parser<'input, T>, name: String) {
		@@ -270,3 +303,6 @@ if let Some(parent) = self.parsers.last() {

		/// Pushes a replay parser onto the stack.
		/// Push a replay parser onto the stack.
		///
		/// Replay parsers are used for included content that has already been parsed into events.
		/// `name` is returned by [`Self::stack`] for diagnostics.
		pub fn push_replay_parser(&mut self, mut parser: ReplayParser<'input>, name: String) {
		@@ -281,3 +317,6 @@ if let Some(parent) = self.parsers.last() {

		/// Pushes a custom parser onto the stack and primes the next event to be returned from it.
		/// Push a custom parser and set the first event that should be returned from it.
		///
		/// This is used when the caller has already consumed the parser's first event before deciding
		/// to place it on the stack.
		pub fn push_custom_parser_with_current(
		@@ -296,3 +335,3 @@ &mut self,

		/// Returns the anchor offset that a newly pushed parser should inherit.
		/// Return the anchor offset that a newly pushed parser should inherit.
		#[must_use]
		@@ -303,3 +342,3 @@ pub fn current_anchor_offset(&self) -> usize {

		/// Returns the names of the parsers currently in the stack.
		/// Return the names of the parsers currently in the stack, from bottom to top.
		#[must_use]
		@@ -364,3 +403,3 @@ pub fn stack(&self) -> Vec<String> {
		self.propagate_anchor_offset_from_popped(&popped);
		return Err(e);
		return e.into_result();
		}
		@@ -372,3 +411,3 @@ Some(Ok((Event::DocumentEnd, span))) => {

		// Check if it has more documents
		// Continue the parent parser if it has more documents.
		let peek_res = match self.parsers.last_mut().unwrap() {
		@@ -434,3 +473,3 @@ AnyParser::String { parser, .. } => parser.peek(),
		}
		Err(e) => Some(Err(e)),
		Err(e) => Some(e.into_result()),
		}
		@@ -457,3 +496,3 @@ }
		}
		Err(e) => Some(Err(e)),
		Err(e) => Some(e.into_result()),
		}
		@@ -468,6 +507,6 @@ }
		while let Some(res) = self.next_event() {
		// Fetch the next event, which is properly synced across the stack
		// Fetch the next event from the active stack entry.
		let (ev, span) = res?;

		// Track if we need to stop based on `multi`
		// Track whether to stop based on `multi`.
		let is_doc_end = matches!(ev, Event::DocumentEnd);
		@@ -482,3 +521,3 @@ let is_stream_end = matches!(ev, Event::StreamEnd);

		// If we only want a single document and we just reached the end of one, stop
		// Stop after one document when multi-document parsing is disabled.
		if !multi && is_doc_end {
		@@ -485,0 +524,0 @@ break;

+64

-5

tests/basic.rs

		#![allow(clippy::bool_assert_comparison)]
		#![allow(clippy::float_cmp)]
		use granit_parser::{Event, Parser, ScalarStyle, ScanError};
		use granit_parser::{Event, Parser, Placement, ScalarStyle, ScanError, StructureStyle};

		@@ -53,2 +53,14 @@ /// Run the parser through the string.

		fn collection_styles(input: &str) -> Vec<(&'static str, StructureStyle)> {
		run_parser(input)
		.unwrap()
		.into_iter()
		.filter_map(\|event\| match event {
		Event::SequenceStart(style, ..) => Some(("sequence", style)),
		Event::MappingStart(style, ..) => Some(("mapping", style)),
		_ => None,
		})
		.collect()
		}

		#[test]
		@@ -74,2 +86,41 @@ fn test_fail() {
		#[test]
		fn test_sequence_structure_styles() {
		assert_eq!(
		collection_styles("- block\n- [flow]\n"),
		vec![
		("sequence", StructureStyle::Block),
		("sequence", StructureStyle::Flow),
		]
		);

		assert_eq!(
		collection_styles("[flow, [nested]]\n"),
		vec![
		("sequence", StructureStyle::Flow),
		("sequence", StructureStyle::Flow),
		]
		);
		}

		#[test]
		fn test_mapping_structure_styles() {
		assert_eq!(
		collection_styles("block:\n child: value\nflow: {child: value}\n"),
		vec![
		("mapping", StructureStyle::Block),
		("mapping", StructureStyle::Block),
		("mapping", StructureStyle::Flow),
		]
		);

		assert_eq!(
		collection_styles("[implicit: flow]\n"),
		vec![
		("sequence", StructureStyle::Flow),
		("mapping", StructureStyle::Flow),
		]
		);
		}

		#[test]
		fn test_empty_doc() {
		@@ -100,3 +151,3 @@ assert_eq!(
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		Event::MappingStart(StructureStyle::Block, 0, None),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		@@ -124,6 +175,10 @@ Event::Scalar("\u{4F60}\u{5273}".into(), ScalarStyle::Plain, 0, None),
		Event::StreamStart,
		Event::Comment(" This is a comment".into(), Placement::Above),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		Event::MappingStart(StructureStyle::Block, 0, None),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		Event::Scalar("b".into(), ScalarStyle::Plain, 0, None),
		Event::Comment(" This is another comment".into(), Placement::Right),
		Event::Comment("#".into(), Placement::Above),
		Event::Comment("".into(), Placement::Above),
		Event::MappingEnd,
		@@ -149,3 +204,3 @@ Event::DocumentEnd,
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::SequenceStart(StructureStyle::Block, 0, None),
		Event::Scalar("plain".into(), ScalarStyle::Plain, 0, None),
		@@ -237,3 +292,5 @@ Event::Scalar("squote".into(), ScalarStyle::SingleQuoted, 0, None),
		Event::StreamStart,
		Event::Comment(" This is a comment".into(), Placement::Above),
		Event::DocumentStart(true),
		Event::Comment("-------".into(), Placement::Right),
		Event::Scalar("foobar".into(), ScalarStyle::Plain, 0, None),
		@@ -263,3 +320,3 @@ Event::DocumentEnd,
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		Event::MappingStart(StructureStyle::Block, 0, None),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		@@ -293,2 +350,3 @@ Event::Scalar("a\n b".into(), ScalarStyle::Literal, 0, None),
		Event::DocumentStart(true),
		Event::Comment("comment".into(), Placement::Right),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		@@ -306,2 +364,3 @@ Event::DocumentEnd,
		Event::Scalar("----".into(), ScalarStyle::Plain, 0, None),
		Event::Comment("comment".into(), Placement::Right),
		Event::DocumentEnd,
		@@ -308,0 +367,0 @@ Event::StreamEnd,

+1

-1

tests/block_scalar_keep.rs

		@@ -21,3 +21,3 @@ // 6FWR: Block Scalar Keep (\|+)
		// "ab\n\n\n"\|Literal\|0\|Span { start: Marker { index: 8, line: 2, col: 1 }, end: Marker { index: 14, line: 5, col: 0 } }
		println!("{:?}\|{:?}\|{:?}\|{:?}", cow, style, size, span);
		println!("{cow:?}\|{style:?}\|{size:?}\|{span:?}");
		}
		@@ -24,0 +24,0 @@ Ok((_event, _)) => {}

+1

-1

tests/closing_bracket.rs

		use granit_parser::{Event, Parser};

		/// Test case 4H7K in yaml_test_suite
		/// Test case 4H7K in `yaml_test_suite`
		#[test]
		@@ -5,0 +5,0 @@ fn misplaced_closing_bracket() {

+1

-1

tests/empty_lines_an_chomping.rs

		@@ -25,3 +25,3 @@ use granit_parser::{Event, Parser};
		Ok((Event::Scalar(cow, style, size, _tag), span)) => {
		println!("{:?}\|{:?}\|{:?}\|{:?}", cow, style, size, span);
		println!("{cow:?}\|{style:?}\|{size:?}\|{span:?}");
		}
		@@ -28,0 +28,0 @@ _ => {}

+17

-0

tests/indentation.rs

		@@ -66,1 +66,18 @@ use granit_parser::{Event, Parser};
		}

		#[test]
		fn queued_key_node_after_comment_keeps_key_indent() {
		let yaml = "? - # key sequence comment\n item\n: value\n";

		let mut key_sequence_indent = None;

		for next in Parser::new_from_str(yaml) {
		let (event, span) = next.expect("valid yaml");
		if matches!(event, Event::SequenceStart(..)) {
		key_sequence_indent = span.indent;
		break;
		}
		}

		assert_eq!(key_sequence_indent, Some(0));
		}

+90

-65

tests/issues.rs

		@@ -1,3 +0,14 @@
		use granit_parser::{Event, Marker, Parser, ScalarStyle, ScanError, Span};
		use granit_parser::{
		Event, Marker, Parser, ScalarStyle, ScanError, Span, StructureStyle,
		StructureStyle::{Block, Flow},
		};

		fn seq(style: StructureStyle) -> Event<'static> {
		Event::SequenceStart(style, 0, None)
		}

		fn map(style: StructureStyle) -> Event<'static> {
		Event::MappingStart(style, 0, None)
		}

		/// Run the parser through the string.
		@@ -99,6 +110,6 @@ ///
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Block),
		map(Block),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		seq(Block),
		Event::Scalar("42".into(), ScalarStyle::Plain, 0, None),
		@@ -112,4 +123,18 @@ Event::SequenceEnd,
		assert_eq!(run_parser(reference).unwrap(), expected);
		assert_eq!(run_parser("[{a: [42]}]").unwrap(), expected);
		assert_eq!(run_parser("[a: [42]]").unwrap(), expected);
		let expected_flow = [
		Event::StreamStart,
		Event::DocumentStart(false),
		seq(Flow),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		seq(Flow),
		Event::Scalar("42".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceEnd,
		Event::MappingEnd,
		Event::SequenceEnd,
		Event::DocumentEnd,
		Event::StreamEnd,
		];
		assert_eq!(run_parser("[{a: [42]}]").unwrap(), expected_flow);
		assert_eq!(run_parser("[a: [42]]").unwrap(), expected_flow);
		assert_eq!(
		@@ -120,6 +145,6 @@ run_parser("[a: [1, 2]]").unwrap(),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		seq(Flow),
		Event::Scalar("1".into(), ScalarStyle::Plain, 0, None),
		@@ -138,6 +163,6 @@ Event::Scalar("2".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("b".into(), ScalarStyle::Plain, 0, None),
		@@ -163,7 +188,7 @@ Event::Scalar("c".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		map(Block),
		seq(Flow),
		map(Flow),
		Event::Scalar("foo".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		seq(Flow),
		Event::Scalar("bar".into(), ScalarStyle::Plain, 0, None),
		@@ -186,4 +211,4 @@ Event::SequenceEnd,
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		@@ -204,7 +229,7 @@ Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		@@ -232,9 +257,9 @@ Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		seq(Flow),
		// No `MappingStart` here.
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("b".into(), ScalarStyle::Plain, 0, None),
		@@ -263,6 +288,6 @@ Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::DoubleQuoted, 0, None),
		Event::SequenceStart(0, None),
		seq(Flow),
		Event::SequenceEnd,
		@@ -284,4 +309,4 @@ Event::MappingEnd,
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		@@ -304,6 +329,6 @@ Event::Scalar("value".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		seq(Flow),
		Event::Scalar("foo".into(), ScalarStyle::Plain, 0, None),
		@@ -323,6 +348,6 @@ Event::SequenceEnd,
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("a".into(), ScalarStyle::Plain, 0, None),
		@@ -343,6 +368,6 @@ Event::Scalar("b".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Flow),
		map(Flow),
		Event::MappingEnd,
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("~".into(), ScalarStyle::Plain, 0, None),
		@@ -365,3 +390,3 @@ Event::Scalar("foo".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("foo".into(), ScalarStyle::DoubleQuoted, 0, None),
		@@ -383,8 +408,8 @@ Event::Scalar("bar".into(), ScalarStyle::DoubleQuoted, 0, None),
		Event::DocumentStart(false),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Block),
		map(Flow),
		Event::Scalar("single line".into(), ScalarStyle::Plain, 0, None),
		Event::Scalar("value".into(), ScalarStyle::Plain, 0, None),
		Event::MappingEnd,
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("multi line".into(), ScalarStyle::Plain, 0, None),
		@@ -407,5 +432,5 @@ Event::Scalar("value".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("k".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Flow),
		Event::Scalar("k".into(), ScalarStyle::Plain, 0, None),
		@@ -428,3 +453,3 @@ Event::Scalar("v".into(), ScalarStyle::Plain, 0, None),
		Event::DocumentStart(true),
		Event::SequenceStart(0, None),
		seq(Block),
		Event::Scalar("a\n".into(), ScalarStyle::Literal, 0, None),
		@@ -476,16 +501,16 @@ Event::SequenceEnd,
		Event::DocumentStart(true),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("array".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Block),
		map(Block),
		Event::Scalar("object".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("array".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Block),
		map(Block),
		Event::Scalar("object".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("array".into(), ScalarStyle::Plain, 0, None),
		Event::SequenceStart(0, None),
		Event::MappingStart(0, None),
		seq(Block),
		map(Block),
		Event::Scalar("text".into(), ScalarStyle::Plain, 0, None),
		@@ -518,3 +543,3 @@ Event::Scalar("Line 1 Line 2".into(), ScalarStyle::Folded, 0, None),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("comment".into(), ScalarStyle::Plain, 0, None),
		@@ -553,3 +578,3 @@ Event::Scalar("hello ... world".into(), ScalarStyle::Plain, 0, None),
		(Event::DocumentStart(true), Span::new(Marker::new(0, 1, 0).with_byte_offset(Some(0)), Marker::new(3, 1, 3).with_byte_offset(Some(3)))),
		(Event::MappingStart(0, None), Span::new(Marker::new(8, 2, 4).with_byte_offset(Some(8)), Marker::new(8, 2, 4).with_byte_offset(Some(8)))),
		(map(Block), Span::new(Marker::new(8, 2, 4).with_byte_offset(Some(8)), Marker::new(8, 2, 4).with_byte_offset(Some(8)))),
		(Event::Scalar("hash_block_null_value".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(8, 2, 4).with_byte_offset(Some(8)), Marker::new(29, 2, 25).with_byte_offset(Some(29))).with_indent(Some(4))),
		@@ -560,3 +585,3 @@
		(Event::Scalar("hash_flow".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(35, 3, 4).with_byte_offset(Some(35)), Marker::new(44, 3, 13).with_byte_offset(Some(44))).with_indent(Some(4))),
		(Event::MappingStart(0, None), Span::new(Marker::new(46, 3, 15).with_byte_offset(Some(46)), Marker::new(47, 3, 16).with_byte_offset(Some(47)))),
		(map(Flow), Span::new(Marker::new(46, 3, 15).with_byte_offset(Some(46)), Marker::new(47, 3, 16).with_byte_offset(Some(47)))),
		(Event::Scalar("hash_flow_null_value".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(47, 3, 16).with_byte_offset(Some(47)), Marker::new(67, 3, 36).with_byte_offset(Some(67)))),
		@@ -566,3 +591,3 @@ (Event::Scalar("null".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(69, 3, 38).with_byte_offset(Some(69)), Marker::new(73, 3, 42).with_byte_offset(Some(73)))),
		(Event::Scalar("array_block_null_value".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(79, 4, 4).with_byte_offset(Some(79)), Marker::new(101, 4, 26).with_byte_offset(Some(101))).with_indent(Some(4))),
		(Event::SequenceStart(0, None), Span::new(Marker::new(109, 5, 6).with_byte_offset(Some(109)), Marker::new(109, 5, 6).with_byte_offset(Some(109)))),
		(seq(Block), Span::new(Marker::new(109, 5, 6).with_byte_offset(Some(109)), Marker::new(109, 5, 6).with_byte_offset(Some(109)))),

		@@ -575,3 +600,3 @@ (Event::Scalar("~".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(110, 5, 7).with_byte_offset(Some(110)), Marker::new(110, 5, 7).with_byte_offset(Some(110)))),
		(Event::Scalar("array_flow_null_value".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(138, 8, 4).with_byte_offset(Some(138)), Marker::new(159, 8, 25).with_byte_offset(Some(159))).with_indent(Some(4))),
		(Event::SequenceStart(0, None), Span::new(Marker::new(161, 8, 27).with_byte_offset(Some(161)), Marker::new(162, 8, 28).with_byte_offset(Some(162)))),
		(seq(Flow), Span::new(Marker::new(161, 8, 27).with_byte_offset(Some(161)), Marker::new(162, 8, 28).with_byte_offset(Some(162)))),
		(Event::Scalar("~".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(162, 8, 28).with_byte_offset(Some(162)), Marker::new(163, 8, 29).with_byte_offset(Some(163)))),
		@@ -581,3 +606,3 @@ (Event::Scalar("null".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(165, 8, 31).with_byte_offset(Some(165)), Marker::new(169, 8, 35).with_byte_offset(Some(169)))),
		(Event::Scalar("indentless_array_block_null_value".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(175, 9, 4).with_byte_offset(Some(175)), Marker::new(208, 9, 37).with_byte_offset(Some(208))).with_indent(Some(4))),
		(Event::SequenceStart(0, None), Span::new(Marker::new(215, 10, 5).with_byte_offset(Some(215)), Marker::new(215, 10, 5).with_byte_offset(Some(215)))),
		(seq(Block), Span::new(Marker::new(215, 10, 5).with_byte_offset(Some(215)), Marker::new(215, 10, 5).with_byte_offset(Some(215)))),

		@@ -608,5 +633,5 @@ (Event::Scalar("~".into(), ScalarStyle::Plain, 0, None), Span::new(Marker::new(215, 10, 5).with_byte_offset(Some(215)), Marker::new(215, 10, 5).with_byte_offset(Some(215)))),
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("hello".into(), ScalarStyle::Plain, 0, None),
		Event::MappingStart(0, None),
		map(Block),
		Event::Scalar("world".into(), ScalarStyle::Plain, 0, None),
		@@ -613,0 +638,0 @@ Event::Scalar(

+25

-7

tests/misplaced_flow_chars.rs

		@@ -1,2 +0,2 @@
		use granit_parser::{Event, Parser, ScalarStyle};
		use granit_parser::{Event, Parser, ScalarStyle, StructureStyle};

		@@ -21,3 +21,6 @@ #[test]
		assert!(matches!(events[1], Event::DocumentStart(false)));
		assert!(matches!(events[2], Event::MappingStart(0, None)));
		assert!(matches!(
		events[2],
		Event::MappingStart(StructureStyle::Block, 0, None)
		));
		if let Event::Scalar(ref v, ScalarStyle::Plain, 0, None) = events[3] {
		@@ -28,3 +31,6 @@ assert_eq!(&**v, "key");
		}
		assert!(matches!(events[4], Event::SequenceStart(0, None)));
		assert!(matches!(
		events[4],
		Event::SequenceStart(StructureStyle::Flow, 0, None)
		));
		assert!(matches!(events[5], Event::SequenceEnd));
		@@ -54,3 +60,6 @@ assert!(matches!(events[6], Event::MappingEnd));
		assert!(matches!(events[1], Event::DocumentStart(false)));
		assert!(matches!(events[2], Event::MappingStart(0, None)));
		assert!(matches!(
		events[2],
		Event::MappingStart(StructureStyle::Block, 0, None)
		));
		if let Event::Scalar(ref v, ScalarStyle::Plain, 0, None) = events[3] {
		@@ -61,3 +70,6 @@ assert_eq!(&**v, "key");
		}
		assert!(matches!(events[4], Event::MappingStart(0, None)));
		assert!(matches!(
		events[4],
		Event::MappingStart(StructureStyle::Flow, 0, None)
		));
		assert!(matches!(events[5], Event::MappingEnd));
		@@ -87,3 +99,6 @@ assert!(matches!(events[6], Event::MappingEnd));
		assert!(matches!(events[1], Event::DocumentStart(false)));
		assert!(matches!(events[2], Event::MappingStart(0, None)));
		assert!(matches!(
		events[2],
		Event::MappingStart(StructureStyle::Block, 0, None)
		));
		if let Event::Scalar(ref v, ScalarStyle::Plain, 0, None) = events[3] {
		@@ -94,3 +109,6 @@ assert_eq!(&**v, "key");
		}
		assert!(matches!(events[4], Event::SequenceStart(0, None)));
		assert!(matches!(
		events[4],
		Event::SequenceStart(StructureStyle::Flow, 0, None)
		));
		if let Event::Scalar(ref v, ScalarStyle::Plain, 0, None) = events[5] {
		@@ -97,0 +115,0 @@ assert_eq!(&**v, "a");

+2

-2

tests/plain_scalar_indicators.rs

		@@ -1,2 +0,2 @@
		use granit_parser::{Event, Parser, ScalarStyle, ScanError};
		use granit_parser::{Event, Parser, ScalarStyle, ScanError, StructureStyle};

		@@ -37,3 +37,3 @@ // Regression guards for StrInput::next_can_be_plain_scalar simplification.
		Event::DocumentStart(false),
		Event::MappingStart(0, None),
		Event::MappingStart(StructureStyle::Block, 0, None),
		Event::Scalar("k".into(), ScalarStyle::Plain, 0, None),
		@@ -40,0 +40,0 @@ Event::Scalar("a:b".into(), ScalarStyle::Plain, 0, None),

+2

-2

tests/plain_tab.rs

		@@ -42,3 +42,3 @@ use granit_parser::{Event, Parser, ScanError};
		match item {
		Ok(_) => continue,
		Ok(_) => {}
		Err(e) => {
		@@ -75,3 +75,3 @@ got_err = Some(e);
		.expect("parser should accept empty explicit-indented block scalar before comment");
		assert_eq!(scalars, vec!["".to_string()]);
		assert_eq!(scalars, vec![String::new()]);
		}

+15

-13

tests/readme.rs

		@@ -1,2 +0,2 @@
		use granit_parser::{Event, Parser, ScanError};
		use granit_parser::{Parser, ScanError};

		@@ -30,18 +30,20 @@ const README: &str = include_str!("../README.md");
		for next in Parser::new_from_str(yaml) {
		let (event, _span) = next?;
		let (event, span) = next?;

		match &event {
		Event::SequenceStart(_, Some(tag)) => {
		lines.push(format!("sequence tag: {}{}", tag.handle, tag.suffix));
		}
		Event::Scalar(value, _, _, Some(tag)) => {
		if let Some(tag) = event.tag() {
		if let Some((value, _style)) = event.scalar() {
		lines.push(format!(
		"scalar tag: {}{} for {value:?}",
		tag.handle, tag.suffix
		"scalar tag: {tag} core-str={} for {value:?}",
		tag.is_yaml_core_schema_tag("str")
		));
		} else if event.is_node() {
		lines.push(format!("node tag: {tag} custom={}", tag.is_custom()));
		}
		_ => {}
		}

		lines.push(format!("{event:?}"));
		lines.push(format!(
		"{event:?} bytes={:?} source={:?}",
		span.byte_range(),
		span.slice(yaml)
		));
		}
		@@ -60,6 +62,6 @@

		println!("Actual output:\n{}", actual);
		println!("Expected output:\n{}", expected);
		println!("Actual output:\n{actual}");
		println!("Expected output:\n{expected}");

		assert_eq!(actual, expected, "README example output is out of sync");
		}

+1

-1

tests/reserved_directive.rs

		@@ -13,3 +13,3 @@ use granit_parser::{Parser, ScanError};
		match item {
		Ok((_event, _span)) => continue,
		Ok((_event, _span)) => {}
		Err(e) => {
		@@ -16,0 +16,0 @@ got_err = Some(e);

+107

-3

tests/span.rs

		@@ -11,4 +11,3 @@ #![allow(clippy::bool_assert_comparison)]
		.nth(char_index)
		.map(\|(byte, _)\| byte)
		.unwrap_or_else(\|\| s.len())
		.map_or_else(\|\| s.len(), \|(byte, _)\| byte)
		}
		@@ -47,3 +46,3 @@
		match x.0 {
		Event::SequenceStart(_, _) => start_stack.push(x.1.start),
		Event::SequenceStart(..) => start_stack.push(x.1.start),
		Event::SequenceEnd => {
		@@ -88,2 +87,81 @@ let start = start_stack.pop().unwrap();
		#[test]
		fn span_slice_returns_source_text_for_valid_byte_ranges() {
		let source = "key: value";
		let span = granit_parser::Span::new(
		Marker::new(5, 1, 5).with_byte_offset(Some(5)),
		Marker::new(10, 1, 10).with_byte_offset(Some(10)),
		);

		assert_eq!(span.slice(source), Some("value"));
		}

		#[test]
		fn span_slice_handles_non_ascii_byte_ranges() {
		let source = "a: 你好";
		let span = granit_parser::Span::new(
		Marker::new(3, 1, 3).with_byte_offset(Some(3)),
		Marker::new(5, 1, 5).with_byte_offset(Some(source.len())),
		);
		let invalid_boundary = granit_parser::Span::new(
		Marker::new(3, 1, 3).with_byte_offset(Some(4)),
		Marker::new(5, 1, 5).with_byte_offset(Some(source.len())),
		);

		assert_eq!(span.slice(source), Some("你好"));
		assert_eq!(invalid_boundary.slice(source), None);
		}

		#[test]
		fn parser_spans_use_byte_offsets_for_non_ascii_input() {
		let source = "a: 你好\nb: c\n";
		let scalars: Vec<_> = Parser::new_from_str(source)
		.filter_map(\|parsed\| {
		let (event, span) = parsed.unwrap();
		if let Event::Scalar(value, ..) = event {
		Some((
		value.into_owned(),
		span.byte_range(),
		span.slice(source).map(std::borrow::ToOwned::to_owned),
		))
		} else {
		None
		}
		})
		.collect();

		assert_eq!(
		scalars,
		vec![
		("a".to_string(), Some(0..1), Some("a".to_string())),
		("你好".to_string(), Some(3..9), Some("你好".to_string())),
		("b".to_string(), Some(10..11), Some("b".to_string())),
		("c".to_string(), Some(13..14), Some("c".to_string())),
		]
		);
		}

		#[test]
		fn span_slice_handles_empty_spans() {
		let source = "key: value";
		let empty = granit_parser::Span::empty(Marker::new(4, 1, 4).with_byte_offset(Some(4)));

		assert_eq!(empty.slice(source), Some(""));
		}

		#[test]
		fn span_slice_returns_none_for_buffered_input_spans_without_byte_offsets() {
		let source = "foo: bar";
		let mut scalar_slices = Vec::new();

		for parsed in Parser::new_from_iter(source.chars()) {
		let (event, span) = parsed.unwrap();
		if matches!(event, Event::Scalar(..)) {
		scalar_slices.push(span.slice(source));
		}
		}

		assert_eq!(scalar_slices, [None, None]);
		}

		#[test]
		fn test_plain() {
		@@ -225,2 +303,28 @@ assert_eq!(
		#[test]
		fn span_slice_for_quoted_scalar_excludes_trailing_comment() {
		let yaml = "key: \"value\" # comment\n";
		let slices: Vec<_> = Parser::new_from_str(yaml)
		.filter_map(\|parsed\| {
		let (event, span) = parsed.unwrap();
		matches!(event, Event::Scalar(..)).then(\|\| span.slice(yaml).unwrap().to_string())
		})
		.collect();

		assert_eq!(slices, vec!["key".to_string(), "\"value\"".to_string()]);
		}

		#[test]
		fn span_slice_for_single_quoted_scalar_excludes_trailing_comment() {
		let yaml = "key: 'value' # comment\n";
		let slices: Vec<_> = Parser::new_from_str(yaml)
		.filter_map(\|parsed\| {
		let (event, span) = parsed.unwrap();
		matches!(event, Event::Scalar(..)).then(\|\| span.slice(yaml).unwrap().to_string())
		})
		.collect();

		assert_eq!(slices, vec!["key".to_string(), "'value'".to_string()]);
		}

		#[test]
		fn test_flow_sequence_explicit_mapping_end_span_order() {
		@@ -227,0 +331,0 @@ let input = "[? a: [b], ? c: &x d, ? e: !t f]";

+25

-0

tests/spec_test.rs

		@@ -94,2 +94,27 @@ #![allow(dead_code)]

		fn str_to_test_error_info(docs: &str) -> String {
		let mut str_error = None;
		let mut iter_error = None;

		for x in Parser::new_from_str(docs) {
		if let Err(e) = x {
		str_error = Some(e);
		break;
		}
		}
		for x in Parser::new_from_iter(docs.chars()) {
		if let Err(e) = x {
		iter_error = Some(e);
		break;
		}
		}

		let str_error = str_error.expect("expected Parser::new_from_str to fail");
		let iter_error = iter_error.expect("expected Parser::new_from_iter to fail");
		assert_eq!(str_error.info(), iter_error.info());
		assert_eq!(str_error.marker(), iter_error.marker());

		str_error.info().to_owned()
		}

		macro_rules! assert_next {
		@@ -96,0 +121,0 @@ ($v:expr, $p:pat) => {

+100

-1

tests/spec_test.rs.inc

		@@ -328,2 +328,73 @@ #[test]
		#[test]
		fn test_ex2_19_integers() {
		let mut v = str_to_test_events(EX2_19).into_iter();
		assert_next!(v, TestEvent::OnDocumentStart);
		assert_next!(v, TestEvent::OnMapStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnMapEnd);
		assert_next!(v, TestEvent::OnDocumentEnd);
		}

		#[test]
		fn test_ex2_20_floating_point() {
		let mut v = str_to_test_events(EX2_20).into_iter();
		assert_next!(v, TestEvent::OnDocumentStart);
		assert_next!(v, TestEvent::OnMapStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnMapEnd);
		assert_next!(v, TestEvent::OnDocumentEnd);
		}

		#[test]
		fn test_ex2_21_miscellaneous() {
		let mut v = str_to_test_events(EX2_21).into_iter();
		assert_next!(v, TestEvent::OnDocumentStart);
		assert_next!(v, TestEvent::OnMapStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnNull);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnSequenceStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnSequenceEnd);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnMapEnd);
		assert_next!(v, TestEvent::OnDocumentEnd);
		}

		#[test]
		fn test_ex2_22_timestamps() {
		let mut v = str_to_test_events(EX2_22).into_iter();
		assert_next!(v, TestEvent::OnDocumentStart);
		assert_next!(v, TestEvent::OnMapStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnMapEnd);
		assert_next!(v, TestEvent::OnDocumentEnd);
		}

		#[test]
		fn test_ex2_23_various_explicit_tags() {
		@@ -616,2 +687,31 @@ let mut v = str_to_test_events(EX2_23).into_iter();
		#[test]
		fn test_ex5_1_byte_order_mark() {
		assert!(str_to_test_events(EX5_1).is_empty());
		}

		#[test]
		fn test_ex5_2_invalid_byte_order_mark() {
		assert_eq!(
		str_to_test_error_info(EX5_2),
		"a BOM must not appear inside a document"
		);
		}

		#[test]
		fn test_ex5_9_directive_indicator() {
		let mut v = str_to_test_events(EX5_9).into_iter();
		assert_next!(v, TestEvent::OnDocumentStart);
		assert_next!(v, TestEvent::OnScalar);
		assert_next!(v, TestEvent::OnDocumentEnd);
		}

		#[test]
		fn test_ex5_10_invalid_use_of_reserved_indicators() {
		assert_eq!(
		str_to_test_error_info(EX5_10),
		"unexpected character: `@'"
		);
		}

		#[test]
		fn test_ex5_11_line_break_characters() {
		@@ -1515,2 +1615,1 @@ let mut v = str_to_test_events(EX5_11).into_iter();
		}

+22

-4

tests/specexamples.rs.inc

		@@ -55,4 +55,16 @@ const EX2_1 : &str =

		// TODO: 2.19 - 2.22 schema tags
		// These examples demonstrate YAML schema resolution in the spec. granit-parser is a low-level
		// event parser, so the tests only assert that the examples parse with the expected node shape.
		const EX2_19 : &str =
		"canonical: 12345\ndecimal: +12345\noctal: 0o14\nhexadecimal: 0xC";

		const EX2_20 : &str =
		"canonical: 1.23015e+3\nexponential: 12.3015e+02\nfixed: 1230.15\nnegative infinity: -.inf\nnot a number: .nan";

		const EX2_21 : &str =
		"null:\nbooleans: [ true, false ]\nstring: '012345'";

		const EX2_22 : &str =
		"canonical: 2001-12-15T02:59:43.1Z\niso8601: 2001-12-14t21:59:43.10-05:00\nspaced: 2001-12-14 21:59:43.10 -5\ndate: 2002-12-14";

		const EX2_23 : &str =
		@@ -76,4 +88,7 @@ "---\nnot-date: !!str 2002-04-28\n\npicture: !!binary \|\n R0lGODlhDAAMAIQAAP//9/X\n 17unp5WZmZgAAAOfn515eXv\n Pz7Y6OjuDg4J+fn5OTk6enp\n 56enmleECcgggoBADs=\n\napplication specific tag: !something \|\n The semantics of the tag\n above may be different for\n different documents.";

		// TODO: 5.1 - 5.2 BOM
		const EX5_1 : &str = "\u{FEFF}# Comment only.";

		const EX5_2 : &str =
		"- Invalid use of BOM\n\u{FEFF}\n- Inside a document.";

		const EX5_3 : &str =
		@@ -96,5 +111,8 @@ "sequence:\n- one\n- two\nmapping:\n ? sky\n : blue\n sea : green";

		// TODO: 5.9 directive
		// TODO: 5.10 reserved indicator
		const EX5_9 : &str =
		"%YAML 1.2\n--- text";

		const EX5_10 : &str =
		"commercial-at: @text\ngrave-accent: `text";

		const EX5_11 : &str =
		@@ -101,0 +119,0 @@ "\|\n Line break (no glyph)\n Line break (glyphed)\n";

+379

-10

tests/test_parser_stack.rs

		@@ -11,4 +11,4 @@ extern crate alloc;
		parser_stack::{ParserStack, ReplayParser},
		BorrowedInput, Event, Marker, Parser, ParserTrait, ScalarStyle, Span, SpannedEventReceiver,
		StrInput, TryEventReceiver, TryLoadError,
		BorrowedInput, Event, Marker, Parser, ParserTrait, Placement, ScalarStyle, ScanError, Span,
		SpannedEventReceiver, StrInput, StructureStyle, TryEventReceiver, TryLoadError,
		};
		@@ -66,6 +66,7 @@
		Event::DocumentEnd => "DocEnd".to_string(),
		Event::Comment(text, _) => alloc::format!("Comment({})", text.as_ref()),
		Event::Scalar(val, _, _, _) => alloc::format!("Scalar({})", val.as_ref()),
		Event::MappingStart(_, _) => "MapStart".to_string(),
		Event::MappingStart(..) => "MapStart".to_string(),
		Event::MappingEnd => "MapEnd".to_string(),
		Event::SequenceStart(_, _) => "SeqStart".to_string(),
		Event::SequenceStart(..) => "SeqStart".to_string(),
		Event::SequenceEnd => "SeqEnd".to_string(),
		@@ -221,3 +222,6 @@ _ => "Other".to_string(),

		let expected_names: Vec<String> = expected.into_iter().map(\|s\| s.to_string()).collect();
		let expected_names: Vec<String> = expected
		.into_iter()
		.map(std::string::ToString::to_string)
		.collect();
		assert_eq!(names, expected_names);
		@@ -269,3 +273,3 @@ }
		}
		Some(Err(e)) => panic!("Parse error: {}", e),
		Some(Err(e)) => panic!("Parse error: {e}"),
		None => break,
		@@ -517,3 +521,3 @@ }
		(Event::DocumentStart(false), span),
		(Event::MappingStart(0, None), span),
		(Event::MappingStart(StructureStyle::Block, 0, None), span),
		(
		@@ -566,3 +570,3 @@ Event::Scalar("k2".into(), granit_parser::ScalarStyle::Plain, 0, None),
		(Event::DocumentStart(false), span),
		(Event::MappingStart(0, None), span),
		(Event::MappingStart(StructureStyle::Block, 0, None), span),
		(
		@@ -608,5 +612,5 @@ Event::Scalar(
		(Event::DocumentStart(false), span),
		(Event::SequenceStart(4, None), span),
		(Event::SequenceStart(StructureStyle::Block, 4, None), span),
		(Event::SequenceEnd, span),
		(Event::MappingStart(7, None), span),
		(Event::MappingStart(StructureStyle::Block, 7, None), span),
		(Event::MappingEnd, span),
		@@ -670,2 +674,139 @@ (Event::DocumentEnd, span),
		#[test]
		fn replay_parser_try_load_single_stops_at_document_end() {
		let span = test_span();
		let replay_events = vec![
		(Event::StreamStart, span),
		(Event::DocumentStart(false), span),
		(plain_scalar("first", 0), span),
		(Event::DocumentEnd, span),
		(Event::DocumentStart(false), span),
		(plain_scalar("second", 0), span),
		(Event::DocumentEnd, span),
		(Event::StreamEnd, span),
		];
		let mut replay = ReplayParser::new(replay_events, 1);
		let mut recv = TryTestReceiver { events: Vec::new() };

		replay.try_load(&mut recv, false).unwrap();

		assert_eq!(
		format_events(&recv.events),
		vec!["StreamStart", "DocStart", "Scalar(first)", "DocEnd"]
		);
		}

		#[test]
		fn replay_parser_try_load_multi_reads_stream_end() {
		let span = test_span();
		let replay_events = vec![
		(Event::StreamStart, span),
		(Event::DocumentStart(false), span),
		(plain_scalar("first", 0), span),
		(Event::DocumentEnd, span),
		(Event::StreamEnd, span),
		];
		let mut replay = ReplayParser::new(replay_events, 1);
		let mut recv = TryTestReceiver { events: Vec::new() };

		replay.try_load(&mut recv, true).unwrap();

		assert_eq!(
		format_events(&recv.events),
		vec![
		"StreamStart",
		"DocStart",
		"Scalar(first)",
		"DocEnd",
		"StreamEnd"
		]
		);
		}

		#[test]
		fn replay_parser_preserves_comment_events() {
		let span = test_span();
		let replay_events = vec![
		(Event::StreamStart, span),
		(Event::Comment(" replay".into(), Placement::Free), span),
		(Event::DocumentStart(false), span),
		(plain_scalar("value", 0), span),
		(Event::DocumentEnd, span),
		(Event::StreamEnd, span),
		];
		let mut replay = ReplayParser::new(replay_events, 1);
		let mut recv = TestReceiver { events: Vec::new() };

		replay.load(&mut recv, true).unwrap();

		assert_eq!(
		format_events(&recv.events),
		vec![
		"StreamStart",
		"Comment( replay)",
		"DocStart",
		"Scalar(value)",
		"DocEnd",
		"StreamEnd"
		]
		);
		}

		#[test]
		fn parser_stack_forwards_comment_events_from_stacked_parsers() {
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(
		Parser::new_from_str("parent: value\n"),
		"parent".to_string(),
		);
		stack.push_str_parser(
		Parser::new_from_str("# child\nchild: value\n"),
		"child".to_string(),
		);

		let events = collect_events(&mut stack).unwrap();
		let names = format_events(&events);

		let child_comment = names
		.iter()
		.position(\|event\| event == "Comment( child)")
		.expect("child comment should be forwarded");
		let child_map = names
		.iter()
		.position(\|event\| event == "MapStart")
		.expect("child mapping should be forwarded");

		assert!(child_comment < child_map);
		}

		#[test]
		fn parser_stack_resolve_preserves_included_comment_events_and_local_spans() {
		let included = "# inc\nincluded: value\n";
		let mut stack: MyStack = ParserStack::new();
		stack.set_resolver(move \|name\| {
		assert_eq!(name, "included.yaml");
		Ok(included.to_string())
		});
		stack.push_str_parser(
		Parser::new_from_str("parent: value\n"),
		"parent".to_string(),
		);
		stack
		.resolve("included.yaml")
		.expect("include should resolve");

		let mut events = Vec::new();
		while let Some(event) = stack.next_event() {
		events.push(event.unwrap());
		}

		let (_, span) = events
		.iter()
		.find(\|(event, _)\| matches!(event, Event::Comment(text, _) if text == " inc"))
		.expect("included comment should be forwarded");

		assert_eq!(span.start.index(), 0);
		assert_eq!(span.end.index(), "# inc".chars().count());
		}

		#[test]
		fn default_empty_stack_peek_and_next_emit_stream_end_once() {
		@@ -695,2 +836,37 @@ let mut stack: MyStack = ParserStack::default();
		#[test]
		fn parser_stack_push_include_resolves_included_content() {
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(Parser::new_from_str("a: 1"), "parent".to_string());
		stack.set_resolver(\|name\| match name {
		"child" => Ok("b: 2".to_string()),
		_ => Err(ScanError::new(
		Marker::new(0, 1, 0),
		"Not found".to_string(),
		)),
		});

		stack.push_include("child").unwrap();

		let events = collect_events(&mut stack).unwrap();

		assert_eq!(
		format_events(&events),
		vec![
		"MapStart",
		"Scalar(b)",
		"Scalar(2)",
		"MapEnd",
		"StreamStart",
		"DocStart",
		"MapStart",
		"Scalar(a)",
		"Scalar(1)",
		"MapEnd",
		"DocEnd",
		"StreamEnd"
		]
		);
		}

		#[test]
		fn iter_parser_inherits_anchor_offset_and_reports_stack() {
		@@ -797,2 +973,30 @@ let mut stack: ParserStack<'static, alloc::vec::IntoIter<char>, StrInput<'static>> =
		#[test]
		fn nested_replay_without_stream_end_is_popped_and_parent_continues() {
		let span = test_span();
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(Parser::new_from_str("parent: value"), "parent".to_string());
		stack.push_replay_parser(
		ReplayParser::new(vec![(plain_scalar("included", 0), span)], 1),
		"replay".to_string(),
		);

		let events = collect_events(&mut stack).unwrap();

		assert_eq!(
		format_events(&events),
		vec![
		"Scalar(included)",
		"StreamStart",
		"DocStart",
		"MapStart",
		"Scalar(parent)",
		"Scalar(value)",
		"MapEnd",
		"DocEnd",
		"StreamEnd"
		]
		);
		}

		#[test]
		fn parser_stack_peek_surfaces_parse_error() {
		@@ -815,1 +1019,166 @@ let mut stack: MyStack = ParserStack::new();
		}

		#[test]
		fn nested_replay_stream_end_is_popped_and_parent_continues() {
		let span = test_span();
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(Parser::new_from_str("parent: value"), "parent".to_string());
		stack.push_replay_parser(
		ReplayParser::new(vec![(Event::StreamEnd, span)], 1),
		"empty".to_string(),
		);

		let events = collect_events(&mut stack).unwrap();

		assert_eq!(
		format_events(&events),
		vec![
		"StreamStart",
		"DocStart",
		"MapStart",
		"Scalar(parent)",
		"Scalar(value)",
		"MapEnd",
		"DocEnd",
		"StreamEnd"
		]
		);
		}

		#[test]
		fn parser_stack_peek_after_stream_end_returns_none() {
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(Parser::new_from_str("a: b"), "p1".to_string());

		while stack.next_event().is_some() {}

		assert!(stack.peek().is_none());
		}

		#[test]
		fn replay_child_propagates_anchor_offset_to_iter_parent() {
		let mut stack: ParserStack<'static, alloc::vec::IntoIter<char>, StrInput<'static>> =
		ParserStack::new();
		let parent = "k1: &a v1\nk3: &c v3"
		.chars()
		.collect::<Vec<_>>()
		.into_iter();
		stack.push_iter_parser(Parser::new_from_iter(parent), "iter-parent".to_string());

		loop {
		let ev = stack.next_event().unwrap().unwrap().0;
		if matches!(ev, Event::Scalar(ref value, _, _, _) if value.as_ref() == "v1") {
		break;
		}
		}

		let span = test_span();
		stack.push_replay_parser(
		ReplayParser::new(
		vec![
		(plain_scalar("included", 2), span),
		(Event::StreamEnd, span),
		],
		1,
		),
		"replay".to_string(),
		);

		let events = collect_events(&mut stack).unwrap();

		assert_eq!(find_anchor_id(&events, "included"), Some(2));
		assert_eq!(find_anchor_id(&events, "v3"), Some(3));
		}

		#[test]
		fn replay_child_propagates_anchor_offset_to_custom_parent() {
		let mut stack: MyStack = ParserStack::new();
		stack.push_custom_parser(
		Parser::new(StrInput::new("k1: &a v1\nk3: &c v3")),
		"custom-parent".to_string(),
		);

		loop {
		let ev = stack.next_event().unwrap().unwrap().0;
		if matches!(ev, Event::Scalar(ref value, _, _, _) if value.as_ref() == "v1") {
		break;
		}
		}

		let span = test_span();
		stack.push_replay_parser(
		ReplayParser::new(
		vec![
		(plain_scalar("included", 2), span),
		(Event::StreamEnd, span),
		],
		1,
		),
		"replay".to_string(),
		);

		let events = collect_events(&mut stack).unwrap();

		assert_eq!(find_anchor_id(&events, "included"), Some(2));
		assert_eq!(find_anchor_id(&events, "v3"), Some(3));
		}

		#[test]
		fn replay_child_propagates_anchor_offset_to_replay_parent() {
		let span = test_span();
		let mut stack: MyStack = ParserStack::new();
		stack.push_replay_parser(
		ReplayParser::new(vec![(plain_scalar("parent", 0), span)], 1),
		"replay-parent".to_string(),
		);
		stack.push_replay_parser(
		ReplayParser::new(
		vec![(plain_scalar("child", 4), span), (Event::StreamEnd, span)],
		1,
		),
		"replay-child".to_string(),
		);

		assert_eq!(
		stack.next_event().unwrap().unwrap().0,
		plain_scalar("child", 4)
		);
		assert_eq!(
		stack.next_event().unwrap().unwrap().0,
		plain_scalar("parent", 0)
		);
		assert_eq!(stack.current_anchor_offset(), 5);
		}

		#[test]
		fn custom_parser_with_current_inherits_parent_anchor_offset() {
		let mut stack: MyStack = ParserStack::new();
		stack.push_str_parser(
		Parser::new_from_str("k1: &a v1\nk3: &c v3"),
		"parent".to_string(),
		);

		loop {
		let ev = stack.next_event().unwrap().unwrap().0;
		if matches!(ev, Event::Scalar(ref value, _, _, _) if value.as_ref() == "v1") {
		break;
		}
		}

		let span = test_span();
		stack.push_custom_parser_with_current(
		Parser::new(StrInput::new("k2: &b v2")),
		"custom".to_string(),
		(plain_scalar("primed", 0), span),
		);

		assert_eq!(stack.current_anchor_offset(), 2);
		assert_eq!(
		stack.next_event().unwrap().unwrap().0,
		plain_scalar("primed", 0)
		);

		let events = collect_events(&mut stack).unwrap();
		assert_eq!(find_anchor_id(&events, "v2"), Some(2));
		assert_eq!(find_anchor_id(&events, "v3"), Some(3));
		}

+4

-4

tests/unclosed_flow.rs

		@@ -18,3 +18,3 @@ use granit_parser::{Event, EventReceiver, Parser};
		println!("Events: {:?}", collector.0);
		println!("Result: {:?}", res);
		println!("Result: {res:?}");
		assert!(res.is_err());
		@@ -34,3 +34,3 @@ let err = res.unwrap_err();
		println!("Events: {:?}", collector.0);
		println!("Result: {:?}", res);
		println!("Result: {res:?}");
		assert!(res.is_err());
		@@ -50,3 +50,3 @@ let err = res.unwrap_err();
		println!("Events: {:?}", collector.0);
		println!("Result: {:?}", res);
		println!("Result: {res:?}");
		assert!(res.is_err());
		@@ -66,3 +66,3 @@ let err = res.unwrap_err();
		println!("Events: {:?}", collector.0);
		println!("Result: {:?}", res);
		println!("Result: {res:?}");
		assert!(res.is_err());
		@@ -69,0 +69,0 @@ let err = res.unwrap_err();

+10

-7

tests/yaml-test-suite.rs

		@@ -363,2 +363,6 @@ use std::{
		fn on_event(&mut self, ev: Event<'input>, span: Span) {
		if matches!(ev, Event::Comment(..) \| Event::Nothing) {
		return;
		}

		if let Some((last_ev, last_span)) = self.last_span.take() {
		@@ -383,3 +387,3 @@ if span.start.index() < last_span.start.index()

		Event::SequenceStart(idx, tag) => {
		Event::SequenceStart(_, idx, tag) => {
		format!("+SEQ{}{}", format_index(idx), format_tag(tag.as_ref()))
		@@ -389,3 +393,3 @@ }

		Event::MappingStart(idx, tag) => {
		Event::MappingStart(_, idx, tag) => {
		format!("+MAP{}{}", format_index(idx), format_tag(tag.as_ref()))
		@@ -411,3 +415,3 @@ }
		Event::Alias(idx) => format!("=ALI *{idx}"),
		Event::Nothing => return,
		Event::Comment(..) \| Event::Nothing => unreachable!("comments are ignored above"),
		};
		@@ -457,7 +461,6 @@ self.events.push(line);
		continue;
		} else {
		Some(format!(
		"line {idx} differs: \n=> expected `{exp}`\n=> found `{act}`",
		))
		}
		Some(format!(
		"line {idx} differs: \n=> expected `{exp}`\n=> found `{act}`",
		))
		}
		@@ -464,0 +467,0 @@ (Some(a), None) => Some(format!("extra actual line: {a:?}")),

+4

-4

tools/README.md

		@@ -27,12 +27,12 @@ # `granit-parser` tools
		↳ DocumentStart
		↳ SequenceStart(0, None)
		↳ MappingStart(0, None)
		↳ SequenceStart(Block, 0, None)
		↳ MappingStart(Block, 0, None)
		↳ Scalar("foo", Plain, 0, None)
		↳ Scalar("bar", Plain, 0, None)
		↳ MappingEnd
		↳ MappingStart(0, None)
		↳ MappingStart(Block, 0, None)
		↳ Scalar("baz", Plain, 0, None)
		↳ Scalar("~", Plain, 0, None)
		↳ Scalar("c", Plain, 0, None)
		↳ SequenceStart(0, None)
		↳ SequenceStart(Flow, 0, None)
		↳ Scalar("3", Plain, 0, None)
		@@ -39,0 +39,0 @@ ↳ Scalar("4", Plain, 0, None)

-62

garden.yaml

		# Use "cargo install garden-tools" to install garden https://gitlab.com/garden-rs/garden
		#
		# usage:
		# garden build
		# garden test
		# garden check
		# garden fmt
		# garden fix

		commands:
		bench: cargo bench "$@"
		build: cargo build "$@"
		check>:
		- check/clippy
		- check/fmt
		- build
		- test
		- check/profile
		check/clippy: cargo clippy --all-targets "$@" -- -D warnings
		check/fmt: cargo fmt --check
		check/profile: \|
		cargo build \
		--profile=release-lto \
		--package gen_large_yaml \
		--bin gen_large_yaml \
		--manifest-path tools/gen_large_yaml/Cargo.toml
		clean: cargo clean "$@"
		coverage: cargo kcov "$@"
		doc: cargo doc --no-deps --package yaml-rust2 "$@"
		ethi/bench: \|
		cargo build --release --all-targets
		cd ../Yaml-rust && cargo build --release --all-targets
		cd ../libfyaml/build && ninja
		cargo bench_compare run_bench
		fix: cargo clippy --all-targets --fix "$@" -- -D warnings
		fmt: cargo fmt "$@"
		test: cargo test "$@"
		update: cargo update "$@"
		watch: cargo watch --shell "garden check"

		trees:
		yaml-rust2:
		description: A pure Rust YAML implementation
		path: ${GARDEN_CONFIG_DIR}
		url: "git@github.com:Ethiraric/yaml-rust2.git"
		remotes:
		davvid: "git@github.com:davvid/yaml-rust2.git"
		yaml-rust: "git@github.com:chyh1990/yaml-rust.git"
		gitconfig:
		# Access yaml-rust2 pull requests as yaml-rust2/pull/*
		remote.yaml-rust2.url: "git@github.com:Ethiraric/yaml-rust2.git"
		remote.yaml-rust2.fetch:
		- "+refs/pull//head:refs/remotes/yaml-rust2/pull/"
		# Access yaml-rust pull requests as yaml-rust/pull/*
		remote.yaml-rust.fetch:
		- "+refs/heads/:refs/remotes/yaml-rust/"
		- "+refs/pull//head:refs/remotes/yaml-rust/pull/"

		yaml-test-suite:
		description: Comprehensive, language independent Test Suite for YAML
		path: tests/yaml-test-suite
		url: https://github.com/yaml/yaml-test-suite

justfile

Sorry, the diff of this file is not supported yet

Cargo.toml.orig

Sorry, the diff of this file is not supported yet

src/parser.rs

Sorry, the diff of this file is too big to display

src/scanner.rs

Sorry, the diff of this file is too big to display

		@@ -21,3 +21,3 @@ // 6FWR: Block Scalar Keep (\|+)
		// "ab\n\n\n"\|Literal\|0\|Span { start: Marker { index: 8, line: 2, col: 1 }, end: Marker { index: 14, line: 5, col: 0 } }
		println!("{:?}\|{:?}\|{:?}\|{:?}", cow, style, size, span);
		println!("{cow:?}\|{style:?}\|{size:?}\|{span:?}");
		}
		@@ -24,0 +24,0 @@ Ok((_event, _)) => {}

granit-parser - cargo Package Compare versions