I went through the my electrical diagrams and there's not a lot that can set off that alarm. My boat is the same era and similar size so I'm guessing your electrical setup is very similar to mine. I also have the same white relay in my main electrical panel. The relay is 120 volts and is connected between the neutral and L1. It does not monitor L2. Basically, when 120 volts is present on L1, it will be energized and open the alarm circuit. It's a normally closed relay contact. It's not connected in any way to the ground and I can't really see how it would set off the alarm if a ground leak was present.
My feeling is that your relay coil is getting weak. It's energized as soon as there is AC power on the boat so it had a long working life. And since you say that the problem is not present when you're running on your generator also makes sense if the relay coil is tired. When you're running on your generator, you get a good 120/240 volts no matter the load. When you're on shore power, you actually never really get that much voltage. Sometimes a lot lower depending on the load on the marina shore power circuit you're on. If you get a lower voltage, and your relay coil is getting tired, you might get the occasional alarm without any visible effect on your AC lights (flickering). Just a thought. I might be wrong but that's my take on your issue.
Also, as others have mentioned before in this tread, you might want to check your L1 fuse near your shore power connection on the boat. Might be intermittent and you wouldn't see your light flicker if they're on L2. These fuses can get pretty hot if you're pulling the full 50 Amps and they have a limited life. I've seen those fail intermittently before. I replace mine once in a while with brand new ones and clean their holder. Personally, I always thought they were more of a fire hazard than a protection... One thing though, if it's not the relay acting on you, the problem seems to be between the shore power pedestal plug and the output of your transformer on your boat. I would check all accessible connections before doing anything else. It's a good thing to do when the boat gets old. Lots of fires start there.
Hope this helps ;-)