Should I be overly concerned about the mold I keep finding in my room?

For over a year I've been finding mold of all colors growing in my room (I've found white, green, blue, grey and even the black kind in my room over the past year) I live in the basement which is always extremely damp, regardless of the season. Mind you I am a very clean person and I clean my room once every week. Last summer I found a surprising amount of black mold growing behind my bed. I have asthma and allergies, but it never seemed to bother me too much. Now I've found 2 very small patches of blackish grey mold on the same wall again. I'm not too worried as I've apparently had mold growing in my room for the past year (at the very least) and haven't had an asthma attack directly caused by it. And my allergies only really bother me outside and such. I am however pretty disgusted by it and plan on removing it as soon as I possibly can. But what I was wondering was, should I be too concerned over it? I mean it's obviously not the greatest thing on the planet for my health, but it doesn't seem like it's really done much damage.