Rust - How to parse UTF-8 alphabetical characters in nom? - parsing

Question

I am trying to parse character sequences of alphabetical characters, including german umlauts (ä ö ü) and other alphabetical characters from the UTF-8 charset.
This is the parser I tried first:
named!(
parse(&'a str) -> Self,
map!(
alpha1,
|s| Self { chars: s.into() }
)
);
But it only works for ASCII alphabetical characters (a-zA-Z).
I tried to perform the parsing char by char:
named!(
parse(&str) -> Self,
map!(
take_while1!(nom::AsChar::is_alpha),
|s| Self { chars: s.into() }
)
);
But this won't even parse "hello", but result in an Incomplete(Size(1)) error:
How do you parse UTF-8 alphabetical characters in nom?
A snippet from my code:
extern crate nom;
#[derive(PartialEq, Debug, Eq, Clone, Hash, Ord, PartialOrd)]
pub struct Word {
chars: String,
}
impl From<&str> for Word {
fn from(s: &str) -> Self {
Self {
chars: s.into(),
}
}
}
use nom::*;
impl Word {
named!(
parse(&str) -> Self,
map!(
take_while1!(nom::AsChar::is_alpha),
|s| Self { chars: s.into() }
)
);
}
#[test]
fn parse_word() {
let words = vec![
"hello",
"Hi",
"aha",
"Mathematik",
"mathematical",
"erfüllen"
];
for word in words {
assert_eq!(Word::parse(word).unwrap().1, Word::from(word));
}
}
When I run this test,
cargo test parse_word
I get:
thread panicked at 'called `Result::unwrap()` on an `Err` value: Incomplete(Size(1))', ...
I know that chars are already UTF-8 encoded in Rust (thank heavens, almighty), but it seems that the nom library is not behaving as I would expect. I am using nom 5.1.0

Answer 1

First nom 5 use function for parsing, I advice to use this form because error message are much better and the code is much cleaner.
You requierement is odd, you could just take the full input make it a string and over:
impl Word {
fn parse(input: &str) -> IResult<&str, Self> {
Ok((
&input[input.len()..],
Self {
chars: input.to_string(),
},
))
}
}
But I guess your purpose is to parse a word, so here a example of what you could do:
#[derive(PartialEq, Debug, Eq, Clone, Hash, Ord, PartialOrd)]
pub struct Word {
chars: String,
}
impl From<&str> for Word {
fn from(s: &str) -> Self {
Self { chars: s.into() }
}
}
use nom::{character::complete::*, combinator::*, multi::*, sequence::*, IResult};
impl Word {
fn parse(input: &str) -> IResult<&str, Self> {
let (input, word) =
delimited(space0, recognize(many1_count(none_of(" \t"))), space0)(input)?;
Ok((
input,
Self {
chars: word.to_string(),
},
))
}
}
#[test]
fn parse_word() {
let words = vec![
"hello",
" Hi",
"aha ",
" Mathematik ",
" mathematical",
"erfüllen ",
];
for word in words {
assert_eq!(Word::parse(word).unwrap().1, Word::from(word.trim()));
}
}
You could also make a custom function that use is_alphabetic() instead of none_of(" \t") but this require make a custom error for nom and is currently in my opinion very annoying to do.

Answer 2

On this Github Issue a fellow contributor quickly whipped up a library (nom-unicode) to handle this nicely:
use nom_unicode::complete::{alphanumeric1};
impl Word {
named!(
parse(&'a str) -> Self,
map!(
alphanumeric1,
|w| Self::new(w)
)
);
}