I have almost the exact strand of synesthesia as you do - colors for numbers and letters, numbers being stronger, and my colors are different than yours. I know about the sense that something is "off" - it's similar to the musician with perfect pitch, some of whom get nauseous if a note is not tuned precisely. One advantage has been doing math in my head very quickly - the colors make it very visual. Socially, I've had more problems as an adult - today's aesthetic is very low contrast, one or two subdued colors, whereas I create multicolored charts from what I envision, but often have to make second versions for certain people. And spreadsheets are too much for me, I sense the vibrating formulae behind the cell. But I do just fine with tables, I don't need the spreadsheet doing calculations anyway. There ought to be a book of synesthetes' stories because I've heard some that were just fantastic.