Nuclear energy only has two significant concerns at this point, assuming that the reactor itself is safely nuilt and well monitored.
A) Waste material - disposing of the unsalvagable material in an environmentaly safe way
B) Safety in the event of a natural dissaster, like what we saw in Fukushima, as there is not that much you can do when a Tsunami Wave is coming or an Earthquake just hit
Other than these, nuclear energy is as close to green energy as it gets. The only thing that goes out of the system is excess heat in the form of steam.
The irrational fear issue is kind of similar to how people fear travelling by plane, even though it is statistically much more likely that you'll die in a car accident. When there's a car crash, even a big one, there's usually 'only' a few casualties, if any. When there's a plane crash, however, the numbers can get up to hundreds at once, which is a much bigger story for the mainstream media to rant about.