When on a picnic, I've been known to toss an apple core or banana peel into the woods. Turns out, even "organic litter" is a bad idea. From Mother Nature Network:
…An apple core can take two months to decompose and a banana peel can take up to two years, by some reports. Although that's a mere blip compared to the estimated decomposition time for plastics — 20 years for a plastic bag, 200 years for a straw or 450 years for a plastic bottle — it's not like these food items will disintegrate quickly.
After watching hikers toss a sandwich on a trail, Marjorie "Slim" Woodruff, who hikes and works in the Grand Canyon, set up a small experiment. She put an apple core, a banana peel, orange peels, chewing gum and tissue paper in a cage of chicken wire, wide enough to allow small animals to go in and out. After six months, the orange peels had dried out, the banana peel had turned black, the chewing gum was the same and the tissue had become a blob. Nothing had been eaten or had rotted.
She buried the same items in sand and soil and six months later everything was still recognizable.
Also, if animals do munch on your edible trash, that can interfere with their own natural foraging while also attracting them to potentially dangerous areas frequented by humans.