Hi tbeahrs,
many thanks for the feedback and that's also a very good question!
With the limited mash tun volume, it isn't just a case of add more grain and everything else stays the same, if you add more grain for the same volume of strike water eventually something has to give and that's head- space, i.e. an overflowing mash! However, we could reduce the strike water volume and increase the grain mass, this indeed can be done, but as you've noted, efficiency goes down the gurgler. There's a few reasons for this, the good thing is the answer lies partly in just simple mathematics (presuming all else is equal). Reduced volume of water = less working fluid to mobilise the sugars, and when a higher
proportion of the original strike water volume remains stuck to the grain (i.e. same amount per kg of grain but lesser amount to begin with = higher proportion), efficiency suffers. There is a minimum proportion of liquor which is never drained from wet grain, more grain equates to more liquor retained, and when that medium for carrying away dissolved sugars get eaten in to, efficiency suffers. Sure, the concentration of dissolved sugars can increase, there could be some beneficial aspects (for efficiency) when it comes to the actual mash dynamics, but not enough to offset those proportional losses, hence overall efficiency is reduced.
This are a few reasons why sparging becomes a handy tool for stove top brewers. Firstly, bearing in mind that Maxi-BIAB uses a more conventional L:G ratio for mashing of around 3-4 L/kg, it frees up those sugars which would otherwise be lost from the process, and does so in a way which is not unfamiliar to brewing, i.e. a simple rinse step. Secondly, it takes advantage of the kettle volume which would otherwise be 'wasted', and as long as your heat source can manage it, I say use it!
When it comes to the actual numbers, IIRC, 5.5kg is the most I've ever mashed in a 19L stockpot, but my usual grainbill is about 4.5kg. This often yields about 23-25L of 1.050-1.055 in the fermenter, although I often supplement this with 5% of the grainbill as sugar, but that is indeed to style. This 4.5kg grainbill seems to be where efficiency is peaking, very seldom do I even have to do the numbers to work out what might have gone wrong. However, it seems 6kg can be managed without too much hassle or bother.
FWIW, some rough sums I did a while back indicated that up to 6kg the relative efficiency loss was tending towards 3% per kg of grain over 4kg, there's a chart below (the 6kg data is not my own, so YMMV). Beyond that, I suspect it will drop considerably, its something I've been meaning to test.
Hope this helps!