I really enjoyed the confusion and interest around that dress. I'm not sure I could draw any qualitative conclusion about my color vision having seen it as blue and black. I think there are a few things at play that are interesting though.
You're right, there's something psychological going on here. I am pretty sure part of it, in the case of the dress, has to do with it being a photograph. There are certain cues and signal our brains have been trained to expect when view a 2-dimension representation of a 3-dimensional object. That picture apparently sends a number of mixed visual cues that get interpreted differently for different people. I'm not sure this phenomenon would occur when viewing the same dress in real life.
As an actual optical device, my understanding of our eyes is that the actual light image that passes through the lenses and hits the retinas (upside-down) is rather crude and is "cleaned up in post" by our brains which flips the image right side up, combined the binocular inputs, fixes distortion, etc. Even this amazing feat is a somewhat fragile system illustrated by the long history of optical illusions.
Which brings me to how we can account for the mild unreliability of what we see, especially in our paintings. I would highly recommend, for those who haven't read it, a copy of Josef Albers book Interaction of Color. As a practicing artist in the Bauhaus School, Albers compiled this very comprehensive work that shows, in detail, how colors behave in relation to each other and how we typically perceive these relationships.