The vast majority of airports don’t have a tower. There may be a CTAF to announce your presence and intentions, but there’s no requirement to have a radio or use one if you have it, and making an announcement doesn’t give you any sort of right-of-way.
But they do have traffic patterns.
The reason for the traffic pattern is so that people are in predictable places, even in the absence of a tower or a radio announcement.
It’s not obvious to me how turns keep you from seeing other traffic, especially from vehicles with as unobstructed a view as hang gliders and paragliders.
I suspect that in many cases, PGs just aren’t looking for/don’t recognize the consequences of a hang glider descending on the southeast corner of the LZ. That guy that’s “way out there” and “not a factor” will shortly be going downwind at above average speeds. That’s not to spite you. That’s the accepted pattern.
I’d also like to point out that the landing pattern is not the place to extend your flight duration. This is true for both HGs that circle directly over the northeast corner of the LZ instead of losing altitude somewhere near the southeast corner as well as the PGs that feel duty-bound to eke every last second of flight out of the light lift over the HG training hill. Land. Getting vertical separation from other people so you’re not landing at the same time is probably another conversation, but that should have been done earlier.