OK, so I looked through the source code (

) and re-implemented the Constellation Modulator using individual blocks and... it worked a lot better.
First: the range on Random Source should indeed be [0,256), as it was in your original version. Each 8-bit byte gets chunked into four 2-bit symbols, so the weird "house" look was because 3 out of every 4 symbols was 00. This makes repeating patterns, which are Bad.
Secondly, the actual problem with the excessively wide signal is because the RRC filter in the Constellation Modulator is producing I/Q values outside of the range (-1,1). When the samples are being converted from complex to int16_t for transmission, an overflow occurs and everything goes bad.
Here it is without scaling the output from the modulator:
https://photos.app.goo.gl/I0MBeFRkVep4IkD02
https://photos.app.goo.gl/sLaQU3dNoVfEUmYt1
Note the numerous points falling outside of (-1,1)
And here it is, multiplying by 0.707 (arbitrary choice of 1/sqrt(2)):
https://photos.app.goo.gl/mx3Z218dkDuBbTS83
https://photos.app.goo.gl/os8aacLSsjvhE74I2
Everything's within (-1,1) and life is good!
Here's my flowgraph for the above (only thing changed was the multiplication constant):
https://photos.app.goo.gl/rIP14a49HatKzOD53
I have reason to believe your receiver will have a better time with this!
Thank you for following up! I need more practice with signal processing, for sure
