Disclaimer: This post has also been published on sirenwatcher.com.
Figure out whether you find native speakers or second language speakers (ESL) the most difficult. As these are totally different issues.
A native speaker is using sounds built into the English language, which can imply there are sounds you will need to learn. Whereas a second language speaker is using sounds from a foreign language you may not use in your native language, merged into their English speech. Such as the German and Indian pronuncation of the W sound (as an English-V sound). You do not need to learn these sounds unless you are dealing with a lot of these speakers on a daily basis.
See this YouTube playlist from a commonly discussed country: India.
Country of origin
Figure out which country is the one you struggle with.
A big issue some foreign language speakers have is in the difference between UK and USA speakers. Other English languages come from that root, such as Australian English is at root British English and based most strongly on the Cockney British accent.
This means if you struggle with British English you may also struggle with Australian or Irish English too. If you struggle with the American accent, you may also struggle with the American accent from the south of the country, or the African-American (AAVE) dialect and accent. Canadian English is a great mixture of American English with British spelling.
This is a great YouTube playlist for some of the variations with tips on different Native English accents.
Some of the origins for a person's listening issues is due to countries focusing on either British or American materials for teaching. In Japan, American English is predominantly taught. In India, it is British English. Therefore Indians may find it harder to understand accents from the US and Japanese may find it harder to understand UK accents. This is not always true but the sounds you have heard can make a huge amount of difference to how comfortable you are using those sounds in your speech.
Origins of all spoken sounds
When you are born you know how to make every sound humans can make. Yet not every language uses every vocal sound. Over time we forget how to use those sounds that our parents have not used around us when we are babies. For the first 5 years of life children mimic their parents and we use these sounds to communicate within our environment. Eventually throwing away the sounds we don't need to use.
Learn to say the sounds your struggling with listening to that may not exist within your host language. This will train your brain to associate the sound with the letters even if you rarely hear them around you. A great method to do this, is by following the International Phonetic Language (IPA) as seen in the video above.
To download the Chart from the video, see this link Wikimedia IPA Chart.
Sounds to go with the Chart can be found here IPA pulmonic consonant chart with audio.
This is a list of all the Phonetic Alphabet sounds in the English language on Wikipedia Help: IPA/English.
A list of all the Phonetic sounds in English can be heard in this YouTube video Interactive Phonetic chart for English Pronunciation.
Phonetics and Phonemes
Figure out what particular sounds you are struggling with listening to.
The next time you notice a word is particularly hard to listen to, ask the person speaking to repeat what they said again. If you still can't understand ask them to spell the word so you can write it down.
Look up the word you have struggled with on thefreedictionary.com or a similar website which shows the phonetic language.
Figure out the particular sounds you commonly struggle with and practice them.
Find materials to practice with.
Some great examples are tongue twisters.
Or lists which provide words by phoneme.
A great source is Rhyme Desk which allows you to search words by phoneme. Pushing one phoneme option and then the button Any, followed by Search will provide you with an extensive list of words containing that one phoneme sound.
Good luck in your advanced English speaking!