Spend 5 minutes in Japan and you'll hear it.
It's the most common form of apology. More polite than "gomen" used with friends and children. Less formal than the complex "moushi wake arimasen."
For a Westerner, it's shocking. Japanese people are always apologising. And when you live in this culture, you're expected to apologise in the same way as well. If you have an accident on your personal time in your personal car, you're expected to apologise to your boss. When you leave work, you apologise to the people you've left behind. Almost any time you walk up to a colleague, your conversation starts with a "sumimasen."
Sometimes, this causes some cultural problems. People are resistant to apologise, especially when it's not their fault. Even more so, when they are not sure there's a fault to be had. Once, I had to apologise for getting to school at 7.32, for a 7.55 departure for a trip!
It's taken me almost 4 years to realise that sumimasen is not about blame. When I first started watching anime, 4 months into my stint in Japan, I only understood a little Japanese. But even with my little, I'd listen to the Japanese, and read the English subtitles, and laugh. That's not what he said! After a while, I remembered the intricacies of subtitling (around 72 spaces and 4-6 seconds per subtitle). I also spent time trying to explain words like "gaman" and "ganbatte" (posts for another time) to friends and family. As opposed to French and Spanish and Italian, where English speakers had the same experiences or often were familiar with them (ex siesta), Japan operated under a completely different set of rules. It is often impossible to translate in a strict "this Japanese word means this English word" sense.
Sumimasen sometimes means "Excuse me," calling someone's attention. But even when it's the apologetic sumimasen, it doesn't always mean you've done something wrong. It may mean that you've done something that puts yourself ahead of your colleagues (bad in Japan's harmonic society). It may mean that something you did inconvenienced someone else, whether or not it's your fault. It may even mean thank you, in the sense of, "oh, you shouldn't have." I later found out that the reason I had to apologise for being "late" for the trip was that there's an unwritten rule, that you get to these things 1/2 hour early, and when I wasn't there, it caused other teachers to worry. (Sadly, this kind of thing happens a lot, because I have no idea of the unwritten cultural rules.)
I hope that gives a little insight into Japanese language and society. If it doesn't, then there's only one thing left to say.