I flew ultralight for seven years, before painfully deciding to sell my kite 2 years ago.
I don't think they are unsafe, if you use just a bit of common sense in maintenance and flying behavior.
The structure is less sturdy than a 'normal' airplane, but the speed and the forces involved are much lower: you cruise at 50-70 mph depending on the type and whatever you do is in that range of speed.
Besides, the safety factor of a decently designed ULM is fairly high.
All the accidents I saw were 'human error' : landing on water with the gear down (Bucaneer), taking off with bad weather, flying low and slow to play smartass etc.
The 90% of problems you may have in a ULM is engine cut-off, not a pleasant thing but if you follow the basic rule 'always keep an emergency landing option in gliding distance' you don't get hurt and in most cases you don't even damage the aircraft. You glide like a butterfly in this things...
Most dangerous things to do : climb too high (the wings can take you to as high as 4000 mt, but your carbs don't have heating and you'll surely have ice), fly in strong winds (you are a butterfly in every meaning), disregard basic maintenance and pre-flight checks, dive too fast to find 'how much can I push it'
On the good side, it is almost impossible to stall (I mean, you really have to work hard if you want to stall it...) and if it happens you recover almost immediately losing very little height (with some exception, like the Quicksilver 500 that is a bit nasty), if you get lost all you need is to find a village and read the road signs or worst case a grass strip with a house nearby: you land and ask for directions...
btw, my crate was a Canadian built Beaver Rx550 with 65Hp Rotax 582 engine. Here is a pic I took in the hangar of the new owner (he contacted me to balance the CG and ailerons after he modified a couple of things)