You need to worry. Think of it this way...
Suppose a non-descript car starts touring around your neighborhood, writing down the address of every house. The people in the car are obviously taking careful notes on each house. And you start to notice that the same people keep coming back to check the neighborhood at different times of the day and night. Would this bother you? It would sure bother me.
Interviews with attackers reveal two important trends: They're willing to invest a lot of time and energy into reconnaissance, and there's a lot of information about a target that can be detected easily. Defenders also need to be aware of the sophistication of modern network scanning tools like nmap. With nmap the attacker can probably determine the operating system of the target, its susceptibility to TCP sequence number prediction, and what services it's running. That’s a lot of information to give away, and allows attackers to choose exploits specific to the targets they want to attack.
That's what these people are doing to your networks. They're checking to see what machines are living on your networks, when the machines are on and off, and which machines respond to which types of mapping attempts. The only reason to collect this type of intelligence about your networks is to exploit the information in some way.
Personally, I don't like being exploited, do you?