Where did the term "yellow-belly" come from?

In a lot of western films and books they call someone who is frightened "yellow" or "yellow belly". We know it means coward but where did that term originate?