Here is a thread about the Y motors running in the opposite direction: Y-Axis motors try to un opposite directions at setup
It still looks like you need to calibrate your machine, though. Here is a thread with a video on how to check (and correct) the calibration of your machine. Have a go with this, and then try carving your square and circle again. Steps/mm Calibration