You can't be too careful with your personal safety. And if you have family or friends who share your dwelling, you can't be too careful with theirs.
That said, the sign you place in your yard needs to have a rider stating that the house will be shown by appointment only and giving a phone number to call. If you can't answer that phone at all times, be sure it has a recorded message assuring people that you will return the call as soon as possible. But don't indicate that you're away at work and won't call until after a certain hour. That just opens you up for burglary.
Even if you are at home all day, don't simply open the door to strangers. If you have children or others who are home alone, instruct them not to open the door unless they know the person standing on the other side.
When you do speak with the caller, get their name and phone number. Then find a reason to call back. You want to at least have some assurance that the person is who he or she claims to be. This may sound over-cautious, but it isn't. That potential buyer may not be interested in the house at all. He could be looking for an easy victim to rob - or worse.
Making potential buyers wait isn't the best selling tactic in the world, but hopefully the flyer in your box will have them interested enough to come back later.
You may have chosen to work with agents and their buyers. If so, it's safe to have the home shown when you're not there, because buyers know that sellers are always instructed to leave during a showing. Just be sure to do due diligence - make sure that agent really is an agent before you hand over a key or a lockbox combination!
Now consider the safety of your possessions. Put away small items of high value that can fit easily into a pocket - and don't leave money, checkbooks, or credit cards in plain view. You should always take this precaution - even when listing with an agent.
When you talk with people coming through your home, don't mention your work schedule or indicate that the house is empty during set hours. In fact, as you'll see when we discuss negotiation, the less you say the better. You do need to answer legitimate questions about the house itself, but don't discuss anything of a personal nature.
I don't want to be an alarmist, but it simply isn't smart to assume that everyone who calls about the house is safe or honest.
Please stay safe.
P.S. You can easily destroy your negotiating position while showing the home - next time I write I'll tell you how to avoid this.