1y ago

Parsing HTML with regex

cross-posted from: https://lemmy.sdf.org/post/12950329

Calligraphy - writing & illuminating & lettering @lemmy.sdf.org

pmk @lemmy.sdf.org

1y ago

Parsing HTML with regex

37 comments

Actually the original is from StackOverflow
- I think it's older than that. Well, parts of it are.
  Though I adore the addition of the fade to chaos in the 2009 Stack Overflow post, I recall seeing the exact early text from that answer linked and re-pasted on older CS forums as early as 2000. It needed posted a lot, shortly after XHTML became popular.
  Part of the contextual humor of the 2009 SO answer, is it takes the trend of the previous answers - getting longer and less coherent each time it needed posted - and extended it to it's logical conclusion.
  I do think the Stack Overflow post is the definitive version.
  I wish I could find the original, or something closer, because we could compare and measure our collective loss of sanity.
- I don't think that using regex to basically do regex stuff on strings that happen to also be HTML really counts as parsing HTML
- Technically, regex can’t pull out every link in an HTML document without potentially pulling fake links.
  Take this example (using curly braces instead of angle brackets, because html is valid markdown):
  
  {template id="link-template"} {a href="javascript:void(0);"}link{/a} {/template}
  That’s perfectly valid HTML, but you wouldn’t want to pull that link out, and POSIX regex can’t really avoid it. At least not with just a single regex. Imagine a link nested within like 3 template tags.
Putting a statement into old calligraphy is nice. But it all boils down to because I said so. If you're going to go to that effort you might as well put the rationale for why it can't possibly parse the language into the explanation rather than because I said so
- The section about "regular language" is the reason. That's not being cheeky, that's a technical term. It immediately dives into some complex set theory stuff but that's the place to start understanding.
  
  English isn't a regular language either. So that means you can't use regex to parse text. But everyone does anyway.
- The text does technically give the reason on the first page:
  It is not a regular language and hence cannot be parsed by regular expressions.
  Here, "regular language" is a technical term, and the statement is correct.
  The text goes on to discuss Perl regexes, which I think are able to parse at least all languages in LL(*). I'm fairly sure that is sufficient to recognize XML, but am not quite certain about HTML5. The WHATWG standard doesn't define HTML5 syntax with a grammar, but with a stateful parsing procedure which defies normal placement in the Chomsky hierarchy.
  This, of course, is the real reason: even if such a regex is technically possible with some regex engines, creating it is extremely exhausting and each time you look into the spec to understand an edge case you suffer 1D6 SAN damage.
- https://en.m.wikipedia.org/wiki/Regular_language
  HTML is famously known for not being a regular language. An explanation isn't required, you can find many formal proofs online (indeed, a junior year CS student should be able to write a proof after their DS/algo/automata classes).
  This very old post is funny because despite it being so famously known as being irregular, stack overflow questions kept popping up asking how to use regular expressions to parse HTML, which you can't do.
- I appreciate the Zalgo calligraphy in particular.
Anyone else read that in the Monty Python holy hand grenade style?
- Xhtml is right out!
- I thought it was a font at first but it looks like actual calligraphy.
  
  It's a personal variant of Foundational script, with uncial and a modernized textura when it starts to derail into zalgo-ness, written with a Lamy Joy 1.9mm nib.
Actually, you can't even parse html (5) with specialized tools or by converting it and then using xml linters (they quit out due to too many errors). Only tools capable of reliably parsing html (mostly) are the big 3 browser engines. Experience from converting saved webpages to asciidoctor, it involves cleaning up manually, despite tidy and pandoc.
- This isn't true. HTML5 made a very strict set of rules and there are a large handful of compliant parsers. But yes, you absolutely can't use an XML parser. You can't even use an XML emitter, as you can emit valid XML that means something completely different in HTML.
  ...what a fucking disaster. I still wish XHTML won.
  
  Real question, why? I feel like there's a story there
Zalgo is Tony the Pony.
- _{he comes}
  he comes
  he comes
  h̵̝̖͂e̵͖͈͐ ̷̛̪̗̜̥̓̉́c̸̜͎̬̦̦̀̀̀͛̚o̴̳͙͂m̷̲̓͋͆́̃ȇ̷̯̞͚̹͗̉͝ͅs̵̹͔̜͕̖̅͗̂͌̀
I am so DOM-ed up that I have to wear a ball-gag
Not knowing it was impossible, he got there and did it!
I tried using regex to parse xmls files once. Then i found this post and realized xml was even worse than html
- XHTML be like "Lemme intorduce myself"
- can anyone recommend a tool to parse xml?
  
  Microsoft transaction SQL version 1 newer than what we have.
  We had to write our own xml parser. And yes it sucks.
  
  XSLT =P
The Chomsky hierarchy is a fucking myth. HTML isn't a formal language but its context free? GTFO, next you'll be telling me the earth isn't flat.