I screen capped the issue I was having notice when I add a colour on the right of the gradient it adds a new swatch after I double click to change it red. When I drag the swatch it adds another new colour under that one again.
I was just testing it and it seems to be an issue when using the gradient tools in Google Chrome. In Safari this issue does not occur.