You can swim anytime after it becomes liquid. The question becomes "how long can you stay in?"
Truthfully, the lake is strange. It will get very warm near shore in July and August. Sometimes it will get into the 80's. Then it will "flip" as the warm water moves offshore and is replaced by cold water and it will be in the 50's or 60's for a couple of days. This is usually after a storm or windy couple of days. Remember, we have depths of over 900ft in Lake Michigan. That will store some pretty cold water all summer long.