# `Yarnspinnex.Parser.Lexer`
[🔗](https://github.com/atavistock/yarnspinnex/blob/main/lib/yarnspinnex/parser/lexer.ex#L1)

Tokenizes a Yarn expression source string. Pure source-to-tokens - no
knowledge of expression grammar or the AST; that lives in
`Yarnspinnex.Parser.Expression`.

# `token`

```elixir
@type token() ::
  {:num, number()}
  | {:str, String.t()}
  | {:bool, boolean()}
  | {:var, String.t()}
  | {:ident, String.t()}
  | {:op, atom()}
  | :lparen
  | :rparen
  | :comma
  | :dot
```

# `tokenize`

```elixir
@spec tokenize(String.t()) :: [token()]
```

Splits expression source into a list of tokens.

---

*Consult [api-reference.md](api-reference.md) for complete listing*