I got this album around the time it came out. I'm a fan of Bob Dylan, but I didn't really connect with this album at first. Jump about 10 years later, and I really like it now. I'm surprised lots of people seem to think of Mississipi as the best song, I like most of the other tracks better, especially Po' Boy, Tweedle Dee & Tweedle Dum, and Moonlight. It's not a folk album, it's not a rock album, it's not quite pop either. I'd say it's more of a rockabilly/country album, and maybe that turns some people off.
I can understand if someone doesn't like his singing. Although his voice has just about disintegrated at this point, he's far from being as out of tune as he can be heard on other albums. I think it's actually one of his better singing albums, within the limits of the quality of his voice of course.
I recommend to give the album a second chance if you've tried it once and not liked it. It has really grown on me.