Commit 5906358 h

Pat Ambrosio · 26-08-25 22:36

Fix nested lists, add strikethrough, stop stray underscore emphasis

Changes: 2 files +167 -3

Changes (Side-by-Side)

Raw

commit 59063589ec9bdb343e1ce47f97aa9b4b46b67584
Author: Pat Ambrosio <cp.ambrosio@gmail.com>
Date:   Tue Aug 25 22:36:34 2026 +0800

    Fix nested lists, add strikethrough, stop stray underscore emphasis
    
    Found by checking whether the mdd corpus (26 files) would survive jenny's
    parser before porting it.
    
    Nested lists were flattened. The nesting machinery was there, but a level
    was peeled per tab or four spaces, while a child is conventionally
    indented to its parent's content column -- two spaces for "- ". The item
    regex tolerates up to three leading spaces as noise, so a two-space child
    matched as a sibling and the list came out flat. Two spaces is now the
    step; a tab remains one level. Written first as an alternation with four
    spaces, a four-space indent matched the wide branch and peeled a single
    level, collapsing three levels into two -- so the step is two spaces and
    four spaces is simply two of them.
    
    A sibling after a nested list opened a second <ul> instead of rejoining
    the first. The closing loop compared against pblock, which is assigned
    nowhere in the program and so was always empty, making the test true for
    any open list: once the walk reached the target depth it kept going and
    closed the outer list as well. Dead state that was harmless only because
    nesting never worked; load-bearing now that it does. It compares against
    nblock, the block kind the line wants, as the rest of the machine does.
    
    Strikethrough was unimplemented -- ~~text~~ passed through as literal
    tildes, 23 times in the corpus. Only the doubled form is markup: a single
    ~ stays literal so "~/path" and "approx ~5" survive, and an unpaired ~~ is
    left alone rather than opening a <del> that runs to the end of the
    document. The first version dropped the rest of the line whenever the
    struck text contained emphasis: nextil shares t1/tag/t2 across frames, so
    parsing the inner text clobbered the tail still waiting in the outer one.
    The link handler keeps its own copy for the same reason.
    
    An underscore that is a tag's whole content is now literal.
    <span class="us">_</span> is a way of writing a visible underscore; it
    opened an <em> that swallowed the following markup, and two of them paired
    with each other across the tags between, emitting mis-nested
    "<b><em></b> ... <b></em></b>". CommonMark reaches the same result through
    its flanking rules. Intraword underscores, real emphasis, bold, and
    emphasis spanning a tag are unaffected.
    
    Verified against the corpus: all 26 files parse, all 23 strikethroughs
    convert, lists nest, and one <em> imbalance remains -- an authoring typo
    in notes/open-questions.md, where "~~**text**~~ *(Resolved...)" leaves a
    fifth asterisk unclosed. That file is not published by the existing site.
    
    Tests: 18 covering both indent widths, three-level nesting, dedent to a
    sibling, flat lists and ordered lists; strikethrough with nested emphasis,
    two per line, single tildes, unpaired markers and tildes in code; and the
    underscore cases including the ones that must keep working. Checked by
    mutation -- reverting each of the four fixes in turn -- all caught. Two
    assertions first used grep -c, which counts matching lines rather than
    occurrences, and passed against correct output for the wrong reason.
    
    Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01WnSAstgnDYdKPwLGv1YZE5

diff --git a/share/jenny/lib/md2html.awk b/share/jenny/lib/md2html.awk
index c214f40..5b0a8c1 100755
--- a/share/jenny/lib/md2html.awk
+++ b/share/jenny/lib/md2html.awk
@@ -58,7 +58,7 @@ function re_quote(s,   out, i, c) {
 }
 
 function nextil(t) {
-	if(!match(t, /[`<&\[*_\\-]|(!\[)|(\[\^)|(\[#)/))
+	if(!match(t, /[`<&\[*_~\\-]|(!\[)|(\[\^)|(\[#)/))
 		return t;
 	t1 = substr(t, 1, RSTART - 1);
 	tag = substr(t, RSTART, RLENGTH);
@@ -263,6 +263,30 @@ function nextil(t) {
 			return t1 "<a href=\"" r "\">" nextil(linktext) "</a>" nextil(pt2);
 		}
 	}
+	# Strikethrough
+	#
+	# Only the doubled form, which is what GFM defines and what authors
+	# write. A single ~ stays literal so that "~/path" and "approx ~5" are
+	# not silently turned into markup. An unpaired "~~" is left alone rather
+	# than opening a <del> that would run to the end of the document.
+	if(tag == "~"){
+		if(sub(/^~/, "", t2)){
+			if(match(t2, /~~/)){
+				strike = substr(t2, 1, RSTART - 1);
+				# t1, tag and t2 are globals shared with every nextil frame,
+				# so parsing the struck text clobbers the tail still waiting
+				# here -- it came back with whatever the inner call left
+				# behind, dropping the rest of the line. The link handler
+				# above keeps its own copy for the same reason.
+				strike_t1 = t1;
+				strike_rest = substr(t2, RSTART + 2);
+				strike_done = nextil(strike);
+				return strike_t1 "<del>" strike_done "</del>" nextil(strike_rest);
+			}
+			return t1 "~~" nextil(t2);
+		}
+		return t1 "~" nextil(t2);
+	}
 	# Emphasis
 	if(match(tag, /[*_]/)){
 		# An underscore between two word characters is literal, not an
@@ -272,6 +296,16 @@ function nextil(t) {
 		# the same exception, and only for _: intraword * still emphasises.
 		if(tag == "_" && match(t1, /[A-Za-z0-9]$/) && match(t2, /^[A-Za-z0-9]/))
 			return t1 tag nextil(t2);
+		# An underscore that is the whole content of an HTML element is
+		# literal: <span class="us">_</span> is a way of writing a visible
+		# underscore, not emphasis. Without this it opened an <em> that
+		# swallowed the markup after it, and two such underscores paired with
+		# each other across the tags between them, emitting mis-nested
+		# "<b><em></b> ... <b></em></b>". CommonMark reaches the same result
+		# by its flanking rules: a delimiter preceded by punctuation and
+		# followed by punctuation is neither opening nor closing.
+		if(tag == "_" && t1 == "" && match(t2, /^</) && stag[ns] != "_" && stag[ns] != "__")
+			return t1 tag nextil(t2);
 		ntag = tag;
 		# `tag` is a literal * or _, so it must be escaped before being
 		# interpolated into a regex: "^" "*" is not a valid regex.
@@ -632,9 +666,21 @@ fence {
 # List and quote blocks
 
 #   Remove indentation
+#
+# One nesting level is peeled per step. A tab or four spaces was the only unit
+# understood here, but a child is conventionally indented to its parent's
+# content column, which for "- " is two spaces -- so two-space nesting, the
+# common form, was invisible: the item regex below tolerates up to three
+# leading spaces as noise, so a two-space child matched as a sibling and the
+# whole list came out flat.
+#
+# Two spaces is the step, not an alternative to four. Written as an
+# alternation with four spaces first, a four-space indent matched the wide
+# branch and peeled a single level, so three levels of four-space nesting
+# collapsed into two. A tab remains one whole level.
 {
 	for(nnl = 0; nnl < nl; nnl++)
-		if((match(block[nnl + 1], /[ou]l/) && !sub(/^(    |	)/, "")) || \
+		if((match(block[nnl + 1], /[ou]l/) && !sub(/^(  |	)/, "")) || \
 		(block[nnl + 1] == "blockquote" && !sub(/^> ?/, "")))
 			break;
 }
@@ -700,11 +746,24 @@ nnl != nl || nblock[nl] != block[nl] {
 	par = (match(b, /[ou]l/)) ? "" : "p";
 }
 nnl < nl || (nnl == nl && nblock[nl] != block[nl]) {
-	for(; nl > nnl || (nnl == nl && pblock[nl] != block[nl]); nl--){
+	# The second clause compares against nblock, the block kind this line
+	# wants, not pblock -- which is assigned nowhere and so was always empty,
+	# making the test true for any open list. Once the walk reached nnl it
+	# therefore kept going and closed the outer <ul> along with the nested one,
+	# leaving a dedented sibling no list to rejoin. Harmless while nesting did
+	# not work at all; load-bearing now that it does.
+	for(; nl > nnl || (nl > 0 && nnl == nl && nblock[nl] != block[nl]); nl--){
 		if(match(block[nl], /[ou]l/))
 			oprint("</li>");
 		oprint("</" block[nl] ">");
 	}
+	# A sibling after a nested list needs a fresh <li> in the list we just came
+	# back to. The "</li><li>" rule earlier only fires when the item is already
+	# at its own depth (nnl == nl), which is false on a dedent.
+	if(newli && nnl > 0 && nnl == nl && match(block[nl], /[ou]l/) && block[nl] == nblock[nl]){
+		oprint("</li><li>");
+		newli = 0;
+	}
 }
 nnl > nl {
 	for(; nl < nnl; nl++){
diff --git a/test/regressions.sh b/test/regressions.sh
index 49218b0..72f767b 100755
--- a/test/regressions.sh
+++ b/test/regressions.sh
@@ -671,6 +671,111 @@ assert_contains "$output" "extensionless"
 it "nothing is written when validation fails"
 assert_file_absent "$ub/.dist/index.html"
 
+suite "nested lists"
+
+# The nesting machinery existed but the indent unit was a tab or four spaces,
+# while a child is conventionally indented to its parent's content column --
+# two spaces for "- ". Two-space nesting, the common form, came out flat.
+
+md_render() {
+  printf '%s\n' "$1" | "$AWK" -f "$JENNY_SHARE_DIR/lib/md2html.awk"
+}
+
+it "a two-space child nests inside its parent"
+out=$(md_render '- a
+  - b
+- c')
+assert_contains "$out" '<li>
+a
+<ul>'
+
+it "a sibling after a nested list rejoins the outer list"
+# The closer walked past the target depth because it compared against pblock,
+# which is assigned nowhere and so was always empty. It closed the outer <ul>
+# too, and the sibling opened a second one.
+assert_eq "$(printf '%s' "$out" | "$GREP" -o '<ul>' | wc -l | tr -d ' ')" "2"
+assert_eq "$(printf '%s' "$out" | "$GREP" -o '</ul>' | wc -l | tr -d ' ')" "2"
+
+it "three levels of two-space nesting all nest"
+deep=$(md_render '- a
+  - b
+    - c
+- d')
+assert_eq "$(printf '%s' "$deep" | "$GREP" -o '<ul>' | wc -l | tr -d ' ')" "3"
+
+it "four-space nesting still works"
+# Two spaces is the step, not an alternative to four: written as an
+# alternation with four first, a four-space indent matched the wide branch and
+# peeled only one level, collapsing three levels into two.
+four=$(md_render '- a
+    - b
+- c')
+assert_eq "$(printf '%s' "$four" | "$GREP" -o '<ul>' | wc -l | tr -d ' ')" "2"
+
+it "a flat list is unaffected"
+flat=$(md_render '- a
+- b
+- c')
+assert_eq "$(printf '%s' "$flat" | "$GREP" -o '<ul>' | wc -l | tr -d ' ')" "1"
+assert_eq "$(printf '%s' "$flat" | "$GREP" -o '<li>' | wc -l | tr -d ' ')" "3"
+
+it "ordered lists nest too"
+onest=$(md_render '1. a
+  1. b
+2. c')
+assert_eq "$(printf '%s' "$onest" | "$GREP" -o '<ol>' | wc -l | tr -d ' ')" "2"
+
+suite "strikethrough"
+
+it "doubled tildes become del"
+assert_contains "$(md_render 'Plain ~~struck~~ here.')" '<del>struck</del>'
+
+it "text after a strike containing emphasis is not lost"
+# nextil shares t1/tag/t2 globally, so parsing the struck text clobbered the
+# tail still waiting in the outer frame -- the rest of the line vanished.
+assert_contains "$(md_render '~~*it*~~ TAIL')" '<del><em>it</em></del> TAIL'
+
+it "two strikes on one line both close"
+two=$(md_render '~~one~~ mid ~~*two*~~ end')
+assert_eq "$(printf '%s' "$two" | "$GREP" -o '<del>' | wc -l | tr -d ' ')" "2"
+assert_contains "$two" 'end'
+
+it "a single tilde stays literal"
+# "~/path" and "approx ~5" must survive untouched.
+assert_contains "$(md_render 'Path ~/x and approx ~5 items.')" '~/x and approx ~5'
+
+it "an unpaired double tilde stays literal"
+# Rather than opening a <del> that runs to the end of the document.
+assert_contains "$(md_render 'Unpaired ~~ stays.')" 'Unpaired ~~ stays.'
+
+it "tildes inside code are untouched"
+assert_contains "$(md_render 'A `~~code~~` span.')" '<code>~~code~~</code>'
+
+suite "underscores in inline HTML"
+
+it "an underscore that is a tag's whole content is literal"
+# <span class="us">_</span> writes a visible underscore. It opened an <em>
+# that swallowed the markup after it.
+assert_contains "$(md_render 'x <span class="us">_</span> y')" '<span class="us">_</span>'
+
+it "two such underscores do not pair across their tags"
+# They emitted mis-nested "<b><em></b> ... <b></em></b>".
+pair=$(md_render 'A <b>_</b> and <b>_</b> here.')
+assert_contains "$pair" '<b>_</b> and <b>_</b>'
+assert_not_contains "$pair" '<em>'
+
+it "real emphasis still works"
+assert_contains "$(md_render 'Real _emphasis_ here.')" '<em>emphasis</em>'
+
+it "bold still works"
+assert_contains "$(md_render 'Real __bold__ here.')" '<strong>bold</strong>'
+
+it "emphasis spanning a tag still works"
+assert_contains "$(md_render 'A _em <b>x</b> more_ end.')" '<em>em <b>x</b> more</em>'
+
+it "intraword underscores are still literal"
+assert_contains "$(md_render 'Use node_modules and snake_case_name.')" 'node_modules and snake_case_name'
+
 suite "MMD metadata end to end"
 
 it "a metadata pair survives into the rendered post"