No language is more easy or difficult to acquire -- for a pre-pubescent learner, they will simply acquire it, provided they have sufficient stimulation during the critical period (deaf children have insufficient stimulation, for example); for adolescents and adults, difficulty will arise in a few key areas (I won't go into all of them or in too much detail though):
Phonemes -- the meaningless individual sounds that make up words (lexemes) -- "cuh", "ah", "tuh", for example: Some languages use phonemes that aren't employed in other languages, that can be, either, difficult for an adult to articulate or difficult for them to distinguish between. For example, it's difficult for most English speaking adults to articulate some of the 'rolling-R's in French, and difficult again for English speaking adults to even distinguish between some of the different bi-labial plosives (Bs and Ps) in Korean. Children, on the other hand, are particularly adept at recognizing minute variations in phonemic constellations. You can look up a study called the Frequent Sucking Phenomenon, which proves this -- babies are language learning machines, and all that babbling and cooing they're doing, that's them learning how to articulate those phonemes that interest them so much.
English is a gender neutral language (for the most part -- there are words like 'host' and 'hostess', for example), unlike like French or German, in which every noun has its grammatical gender which impacts the modifications that the nouns undergo (morphology) and the words in grammatical relation to them -- EINen hund, EINe katze und EIN handy, for example. English speaking adults may find it difficult to retain the grammatical genders for words on top of retaining vocabulary.
As you can already start to see, there is no more inherently "difficult" language to learn, some aspects of language acquisition will come more easily to speakers of one language than another. The real difficulty in learning a new language, which lies in ALL languages, is building a brand new cognitive framework encompassing all the novel syntactical structures (past, present, future, etc.), the morphology, the idioms (nonsensical structures, like "go fly a kite," that, when translated literally, might confuse a non-native speaker) and so on and so on.