Speaking English is not too difficult, speaking English proficiently is another story. So should you be worried about your level of English? That depends on what you are wanting to do with what you are wanting to learn. As a native English teacher, I have a slight advantage over most ESL students simply because I grew up speaking English without many difficulties. I don't remember all too well, the first time my mother and father corrected me when I misused the past simple tense of the word eat. It's "ate" by the way.
There are many reasons to learn English. Perhaps you want to simply communicate with people so you can share ideas and experience the world in a different way. Perhaps you are learning English for educational or work placement opportunities. Maybe you enjoy watching films and you simply don't want to rely on reading subtitles. For many, learning English might stem from the desire to communicate with loved ones, who for one of many reasons, may only speak English. For some, they might simply want to sharpen their mind and find language study a great way to keep sharp.
Whatever the reason, I want to encourage you to pursue any one of these reasons, and do so knowing that I and many other teachers don't expect you to be perfect. If you enjoy meeting new people, take steps to do so. Don't let your fluency or lack thereof be a stumbling block to you. Certainly, there are some individuals who might laugh at you as you stumble through a sentence, but more often than not, it's not intended to belittle you.
Your accent is not a problem for most people if they are listening and are willing to extend some grace to you. Sure, you might say "lice" instead of "rice" but that's okay. Even if you are in a professional setting, the fact that you are communicating in English instead of your native language says a lot. Your accent is a part of you. Yes, work to improve it, but don't let it hinder you. Be yourself. Remember, there are many people out there who want to help you learn and enjoy English, find those people, and let the learning really begin.