Yes, it's underneath the screw-on cover, indeed. I think it's 4 or 5 screws and a couple of them are at a funky angle so it's difficult to get a bite on them with a screwdriver. But you can do it.
If you want to save a buck or three, you can get a used relay from Potomac or off of eBay for probably $5-10, or from a junkyard. I believe it just has to be from a later (1992-1995) model year 124, but I'd have to look it up to make sure. May be the same for all years. In any case, late 124s are easy enough to find in junkyards that that may be the route to go.
The symptoms tend to be non-functioning flashers (corners on all the time, not flashing), intermittent blinker (sometimes blinking fast, slow, or not at all). Like I said, I've had these relays go out on at least 3 124s (including my E500) that I've owned as well as the 1995 E320 sedan that a colleague of mine owns here in TX. Very common and a plug and play fix.
Good luck!
Cheers,
Gerry
DO NOT buy KAE / Kaehler brand relay if you do go new. They are junk!
Here is a better-quality Wehrle relay from Germany:
Here is a Hella brand relay: