CSS: Add hash token type flag

[sirreal]

Summary

Adds CSS Syntax hash-token type flag support to CSSProcessor.

Hash tokens now expose whether they are id or unrestricted via new constants and get_token_type_flag(), matching the spec more closely and giving downstream consumers the information they need to distinguish hash-token behavior.

Exposing that flag lets callers distinguish ID-selector-style hashes.

Testing

Adds coverage for hash-token type flags, delimiter fallback, and clearing the flag between tokens.

The token type flag can likely be re-used for the number type, see
#233.

[sirreal]

I've aligned this with #233, I'd like to land them more or less together.

[sirreal]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

[dmsnell]

is the set of values static and small? could we add them here instead of using string? or as a list in the docblock?

[sirreal]

Yes, this overlaps with #233 here and I avoided writing it all out here.

For now it's just the hash flags: id or unrestricted. #233 adds 2 more.

[dmsnell]

Hash tokens now expose whether they are id or unrestricted

can you give a brief explanation of what this is?

[sirreal]

can you give a brief explanation of what this is?

Goodness, I thought I documented this a bit better. #233 is more like what I had in mind with regards to documentation, the two are very linked (they'll populate the same property and expose the same method to exposed the token flags).

This implements this part of the CSS spec::

hash tokens have a type flag set to either "id" or "unrestricted"

I believe it was necessary when using this tokenizer for selector parsing.

(#233 handles the adjacent " and additionally have a type flag set to either 'integer' or 'number'" part of the spec)

[dmsnell]

so this is like get_token_type()? but only if the type is hash? and what I was getting at, is what is the meaning of id and unrestricted and why would calling code care?

even in the linked spec, I don’t see any obvious explanation for setting that flag or what it means, other than an auxiliary note that an unrestricted hash token “may not need as much escaping.”

[sirreal]

so this is like get_token_type()? but only if the type is hash?

It's like a sub-type of some tokens. Hash tokens have a flag. Number and dimension tokens as well. It conveys some additional information about the token.

Only id-flagged hash tokens are valid ID selectors, so when parsing selectors from the token stream you need to know whether you have an id or unrestricted hash token. It's part of the grammar.

In <id-selector>, the <hash-token>’s value must be an identifier.

If you don't have this information from the token stream, you cannot reliably parse selectors and would need to re-parse the hash token.

[dmsnell]

thanks @sirreal for the extra link. is it right then that this is more or less saying there are valid hash tokens, and there are ignored hash tokens? as in, #\ is not valid in a selector because no identifier could be the space character?

[sirreal]

#000 is an example of an "unrestricted" hash token that's a relevant token and valid CSS, just not in the context of selectors:

#ident {
  color: #000;
}

(In the case of #\ , that is a valid id selector! I believe because the escape sequence can start an ident.)

[dmsnell]

aha, that makes so much more sense. thanks @sirreal for clarifying that. a tiny example comment would make that so much more readily-apparent when reviewing the API