I gave temporary gaurdianship of my son to someone who i umed to be a friend?

he was almost 3 when i did this now he just turned 4 i have been in his life the entire time nothing has been filed in court and i am trying to get him back but they are refusing is this illegal for them to try and keep him from me should i just be able to go get him with no problems should i maybe get children youth and families involved or the police somebody please help im confused and i want my son back the only thing that has been done is a notarized paper which was typed up on our home computer nothing in court