No evidence for this but I feel like about half the departed that you are supposed to wear a poppy for would not care if you didn't wear one.
The tradition itself exists to reinforce the belief that whatever military campaign your nation is engaged is should be unconditionally supported which is nonsense. At the same time I have never seen a strong and collective oppositon to it. So I am loathe to suggest a vocal rejection of it outright. Just do that you can do.