Step one in this case is to upgrade to 11.5 to include your agents and the latest version of Oracle and then implement your OS changeover.
Most of the data is stored in the Oracle Database such that it does not require a large amount of effort with respect to your detection servers. Unless you have made some customizations such as adding DB providers for SQL scanning then the detection servers can be replaced outright by building their replacements and changing the network address to the new systems in the console.
For your endpoint this is slightly more complicated since you have agents directed to that server. If you stand up your 11.5 server you can move the agents from the 10.5 server to 11.5 server once you have 11.x implemented.
For your enforce and DB I would consult the upgrade guide included in the DLP documentation for how to upgrade Oracle and switch bitness if required. The admin guide also includes a migration guide which is essentually doing a cold backup and then a restore to the new server