Annatto comes with its own special flavor, which is why it's also used as a spice. Take a whiff of annatto and you'll be greeted by a nutty, floral aroma. The flavor of annatto is best described as mildly sweet and spicy, with some earthy, musky notes.