Frank makes some very valid points, notably the three orders of magnitude higher power run by transponders and the fact that the antenna (external!) is looking at the ground. 868MHz is only marginally better at getting through "stuff" than 1090MHz, so most of the gain has to be made elsewhere, either by power or by antenna design/location. Power is severely limited by license conditions, so the only option available to us is to improve the all-round visibility of the antenna and (perhaps) its gain.
A dipole is quite small (span about 17cm (6.5")) and it's easy to construct. A ground plane antenna is nearly as good and only half that size but needs metalwork to work against. The real problem is where to put it, so it can see all around. Atop the tail fin is fairly good, as is a composite wing tip but both require lengthy coaxial cable runs, potentially losing some of the benefit through cable losses. A ground plane antenna on top and one underneath the fuselage would work very well but is difficult to engineer and to integrate (power splitters and so on).
If we accept that most risks come from straight ahead give or take 30 degrees then we can get away with something in the cockpit, e.g. on the coaming, but that will be severely compromised in other directions, especially behind and below. It is something of an intractable problem and not something we have had to worry about in the past with other types of avionics. If this was a simple problem we'd have fixed it by now!
