You're knitting a sweater for yourself. Congratulations! You have plenty of yarn in the same dye lot and you even swatched to ensure proper gauge. You've adjusted your needle size, and you're pretty darn proud of yourself. Way to go!
However, you are not a size XS, so casting on only enough stitches for that size and knitting 5" in a 1x1 rib won't get you very far in owning a sweater you can wear.
I suppose I should offer up some kind of congratulations for realizing your mistake now, and not after knitting the entire back. Unfortunately, it means you've officially accomplished nothing in the past 4 days.