Prepare yourself for some serious jealousy acts along the way, SH.
I think everyone loses it from time to time. And I also think that also he was scared, he saw you had some limits also. YOu cannot always be the perfect parent.
Talk to him a lot. During the day, as much as you can. Helps develop some great bounds. My mom was even deciding which food to cook with me
. I was also around 4.
Bottom line: stop beating yourself up! You just have to make it through the day and love him a lot. That's all.
Tell us how it works out after picking him from school.