I would make him take it back and have him apologize to the store manager if you haven't already. I wouldnt make a huge deal out of it because of how young your child is,but I would show him how it would feel if someone took something from him or her,I'm sure your child will understand and not do it again.
I would make her go back in the store and talk to the person in charge at that time and make her say she was sorry for taking it. Then have her put it back where she got it from. Telling her it is not okay to steal and explan why. That if she wants somethng she needs to ask. Haveing her go into a store and tell someone she is sorry is punishment enough at the age of four. Anything bigger then that they wont know what is going on or why somethng happen. Once they leave the stroe most times they wont rememeber what happen.
I would probably confiscate whatever they took, call the store ahead of time, tell the manager that I want to go back to the store and have my child return it and let him/her know what lesson I want my child to learn.
I wouldn't "punish" him/her because they aren't yet old enough to honestly know that what they did was wrong. I would explain that it wasn't okay because we didn't pay for it, and that's called stealing. Then we would take it back to the store. If we couldn't return it for some reason, then we would pay for the item and go home to complete some tasks to pay for the item that Mommy had to use money to pay for.