Don't feel ashamed. So, you went back to him briefly but nothing permanent happened. You kept your virginity intact and you learned a lesson. If you have to hide a relationship from loved ones there is a good chance the relationship is bad news.
You deserve a healthy relationship in which you are respected. He doesn't respect you.