A lot of writing advice will tell you how to edit down at the prose level. Things like getting rid of adverbs: “That always annoyed me,” Yoon said irritably. In this case I would in fact get rid of the adverb because it’s not adding information; irritation is implied by the (rather on the nose) dialogue. On the other hand, if you had a line like “That always annoyed me,” Yoon said dreamily, the adverb is telling you something contrary to expectation, and could reasonably be kept.
Yoon Ha Lee on adverbs.
This is, quite honestly, the best and most concise way I’ve ever heard the whole not-adjectives-except-sometimes-adjectives rule explained.