FHIR-55473 @Josh Mandel recorded this issue and wanting to get some feedback from others in the community. The specific ask is if non-breaking space could be included in the whitespace lexical category .

For whitespace you could consider Unicode Whitespace and Pattern_Whitespace categories. But what is the usecase for string equivalence? It is problematic and inconsistent between existing implementations. You can check ('İ' ~ 'I').combine('İ' ~ 'i').combine('İ' ~ 'ı').combine('I' ~ 'i').combine('I' ~ 'ı').combine('i' ~ 'ı') ('ẞ' ~ 'ß').combine('ẞ' ~ 'SS').combine('ẞ' ~ 'ss').combine('ß' ~ 'SS').combine('ß' ~ 'ss') // Or with escape syntax ('\u0130' ~ 'I').combine('\u0130' ~ 'i').combine('\u0130' ~ '\u0131').combine('I' ~ 'i').combine('I' ~ '\u0131').combine('i' ~ '\u0131') ('\u1e9e' ~ '\u00df').combine('\u1e9e' ~ 'SS').combine('\u1e9e' ~ 'ss').combine('\u00df' ~ 'SS').combine('\u00df' ~ 'ss')

/poll Should we add the non-breaking space (\u00A0) to the list of whitespace lexical elements Yes No

Why "no"?

I think if you want to add non-ASCII whitespace, you should add the whole Unicode category, not just one character.

Add it to the list of characters that are interpreted as a whitespace?

And can are normalised when doing a string equivalence test? (which is what we're discussing)

To the list of characters interpreted as equivalent inside strings. But not to the list of lexical whitespaces. You can see the list of these characters here: https://en.wikipedia.org/wiki/Whitespace_character#Unicode

So you're not saying no, you're saying include these others too?

Yes

Cool, that makes more sense. Thanks.

@Bryn Rhodes thoughts? Concern on the performance impact of adding 20 more characters here?

I'd advocate yes for these reasons: The current 4-character set is ASCII-era thinking NBSP is the most common "invisible surprise" character in real clinical data Implementations using standard Unicode whitespace handling are likely already treating it as whitespace - the spec should align with reality rather than force implementers to write custom matchers At minimum, add \u{00A0}; ideally, consider adopting Unicode's \p{White_Space} property to future-proof it - https://www.unicode.org/reports/tr44/#White_Space

Thanks, that last reference is what I was hunting for!

Maybe this set? https://www.unicode.org/charts/collation/chart_Whitespace.html dotnet has 17 of them listed: https://learn.microsoft.com/en-us/dotnet/api/system.char.iswhitespace?view=net-10.0

While wandering down this rabbit hole, I also came across this: Unicode Normalization - https://unicode.org/reports/tr15 Which helps normalize unicode characters, and lots of programming languages support this too. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize

This is where the unicode whitespace characters are enumerated. https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt

/poll Should we change the whitespace equivalence handling for strings: No Add JUST non-breaking space (\u00A0) Add ALL unicode whitespace (as defined in https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt )

/poll Should we also add unicode normalization to the string equivalence No Add Unicode Normalization (as a may) Add Unicode Normalization

I don't believe that CQL does any of this either.

I think the most universal is collation-based equality with root search collation, normalization, primary strength, and shifted alternate. https://icu4c-demos.unicode.org/icu-bin/collation.html