So I'm trying to get OS X 10.4-10.6 working on QEMU/KVM with full GPU passthrough. I've now got each of these booting inside QEMU/KVM utilizing OpenCore, now it's just a matter of getting the GPUs working.
Right now I'm primarily using an ATI Radeon HD 2400 XT, which has been flashed to be a 2400 Pro for greater compatibility with Mac OS. I was originally using an ATI Radeon HD 3870, which I'd managed to get partially working before swapping it out, but was interested in how common the 2400 XT/Pro was (right now several are available on eBay for only $5) and how accessible that would make this process for people (also it's single-slot which makes it easier to fit in a lot of cases), so that's become my focus for the time being.
Technically none of this research is "new" - people did manage to get stuff like this working back in 2007-2008 when it was more timely to do so - however much of that information seems to have been lost to the sands of time. While InsanelyMac, one of the biggest Hackintosh forums of that era, is thankfully still up and contains a lot of useful information, many of the links on it are to long-gone websites like RapidShare, Megaupload, or dead personal websites. Additionally, open source was not as common as it is today, so many community-made kexts have no documentation on exactly what they do or how they work. As such, a lot of this information has had to be pieced together from many different sources.
Even though the ATI Radeon 2400 XT shipped with a real iMac that supported 10.4 onwards (iMac7,1 which I'm actually using as my board ID for Hackintosh), that GPU was configured differently from its PC counterpart, so it can't just work out of the box. Even (perhaps especially) on a Mac Pro, the PC version of this card won't work out of the box.
It appears graphics support can generally be split into two categories: the "framebuffer", which enables full resolution and multi-monitor support/switching; and QE/CI (Quartz Extreme/Core Image) which enables hardware 3D acceleration. The framebuffer is crucial, since it enables the most basic graphics functionality (and 3D acceleration cannot operate without it), however for full support we need both.
Unfortunately, the framebuffer is also the most challenging. While you might think 3D acceleration would be the hardest to get working, for the most part enabling it has come down to adding my card's device ID to the 3D acceleration kext. The framebuffer seems to be much more complex.
Mac OS X 10.4 and 10.5
In early versions of Mac OS X (namely 10.4 and 10.5), the ATI framebuffer appears to be enabled through kexts called ATINDRV and ATIRNDRV (for earlier and later models respectively). ATI cards also appear to have "shark names" that correspond to different models, each of which has a plugin in the aforementioned kexts:
Injector kexts exist in order to connect GPUs to these plugins, such as Natit and ATIinject. I've tested these, and while in some cases I may have to correct which framebuffer plugin to inject (my card only works with Megalodon, while some sources/injectors claim the 2400 should be Iago), they do indeed seem to work.
...For the most part. For some reason, I get very noticeable flickering (possible seizure warning):
The flickering is definitely a framebuffer issue. In fact, it's not dissimilar to the flickering I experienced on my Surface Laptop 3 Hackintosh, which may be a clue to fix this, but I can't port the exact fix because that was exclusive to both Intel GPUs and WhateverGreen, neither of which I'm using here.
The 3D acceleration kexts are ATIRadeonX1000 and ATIRadeonX2000, and I got 3D working on this card simply by adding my GPU's device/vendor ID after flashing (0x94c31002) to ATIRadeonX2000's Info.plist. As long as the framebuffer is correct, this seems to work - the ripple effect in the above video indicates correct 3D. The flickering is no different with or without 3D enabled, so it's very clearly a framebuffer issue. However, if there's no framebuffer (or perhaps a too-broken framebuffer), enabling 3D results in nothing but a garbled display.
Currently this is the situation for 10.4 and 10.5. Full resolution switching (using Natit) and 3D acceleration (using a manually patched Info.plist in ATIRadeonX2000), but bizarre flickering in the framebuffer that I haven't yet figured out. Presumably Natit is doing something not quite right,
There are a handful of parameters in Natit's Info.plist, but they're all either numbers or Base64 sequences that don't mean anything to me just yet.
(Note: Interestingly I managed to get full GPU support using a Leopard distro on my HD 3870, but never got it working on a vanilla setup before swapping out the GPUs. Perhaps my current setup my work with the HD 3870, but I haven't tried it yet. The Leopard distro did not work out of the box with the HD 2400, possibly because it too was trying to inject Iago rather than Megalodon.)
Mac OS X 10.6 and above
In 10.6 and onwards, the ATI framebuffer code appears to have been rewritten. As such, Natit and ATIinject no longer work out of the box (in fact they appear to cause kernel panics), and the kexts in question have changed. There's no ATINDRV or ATIRNDRV, instead there's ATIFramebuffer, ATISupport, and various ATIxx00Controllers:
The "shark names" including Megalodon appear to be rolled into ATIFramebuffer, and I have every reason to believe if it was injected correctly, it would work, but clearly it's too different for the 10.4/10.5 injectors.
Today, the main solution for framebuffer patching is with a kext known as WhateverGreen. Unfortunately unlike other modern components like OpenCore/VirtualSMC/Lilu, WhateverGreen only supports as early as 10.6, and my guess is this is because of the apparent framebuffer rewrite that seems to have happened in Mac OS.
Unfortunately, while I've had great luck with more modern AMD and Intel cards, the HD 2400 is not fixed out of the box by WhateverGreen. It appears some degree of manual overriding is going to be necessary, which isn't extremely well documented. WhateverGreen is at least open source and contains "sample SSDTs", but like the numbers and Base64 in Natit, all of this framebuffer stuff is well outside my knowledge right now until I can spend some more time to figure it out.
I've also installed 10.7 and 10.8 with this OpenCore configuration, and they seem to work/not work the same way.
The 3D acceleration kexts appear to be the same - ATIRadeonX1000 and ATIRadeonX2000.
(Note: I was at least able to achieve framebuffer functionality with my HD 3870 on 10.6, using Chameleon which has a built in framebuffer injector, however this too does not work on the HD 2400, and I was never able to achieve 3D acceleration - the display would garble, which I think indicates another framebuffer issue, but I couldn't say for sure.)
I think I'm really close to making this work, but I don't think there are any easy answers anymore - I'm going to have to figure out how real framebuffer patching works. Frustratingly, we have current methods that are so close to working, but are just not quite fully correct, which is probably why the more manual WhateverGreen process is recommended now over "easy" solutions like Natit.
To an extent, it seems like we'll have to work it out two separate times: once for 10.6+ and once for 10.4/10.5. The core information will almost certainly be transferable, but the methods will probably have to be different. It may be worth tweaking some of those Natit settings just to see what happens. Unfortunately, I can't find any existing information about getting the HD 2400 working with WhateverGreen, and there's a chance it may not be possible at all since WhateverGreen doesn't guarantee functionality with non-UEFI BIOSes (while it's a virtual machine that could have UEFI, HD 2400 has no support for it, forcing me to use BIOS booting).
So essentially, I think the only step forward is to get into the real nitty gritty of it. I will update here if/when I find any more information.