I opened a case with support and have tried WinPE, but I'm still getting the same 'memory allocation failed' error. I'm currently waiting for a call back from support. Have you opened a case with support?
Since you're getting the same error, it's looking more like a specific issue with the Dell 760s. I've tried DOS, Linux, and Windows, so it doesn't seem like just a driver issue and KB 35816 doesn't help...
rdlog:
Error description:
Memory allocation failed.
(Note: Current volume is OEM from source slot 1.)
Possible causes:
Out of memory.
Possible resolutions:
Install more memory.
Delete some files on the computer you're imaging.
==================== Technical details ====================
Logfile = created Thu Jul 23 08:11:28 2009
Build = RdeployT.exe 6.9 (9024)
Cmdline = F:\RDeploy\Windows\RdeployT.exe -noprompt -mu -f.\Images\d_drive\dell760\winpe\winpe.img -cspeed -split:2040 -dsstatus:10.10.10.3:402 -mcint:10.52.2.28
Status = 5 (0x5)
Source file = imglib\common\abuffer.cpp
Line number = 183 (0xb7)
Stack trace = 0x4c7f6c 0x4c7fb7 0x4af4c4 0x484cd1 0x487642 0x4149ef 0x40352a 0x412294 0x401db0 0x40e42d &Known=0x433e40
Note = Current volume is OEM from source slot 1.
Imaging library revision = 9024 (win32-x86-release build, Thu Jan 29 16:47:10 2009)