oSpy and JIT x86 machine code generation

I was faced with a new and interesting challenge the other day, I had to use oSpy2 to intercept calls to DeviceIoControl. Off I went and added the required bits to oSpyAgent’s config.xml so that it essentially contained:

And doing a quick capture resulted in events like this:

And boom, the application being monitored crashed… The reason why is quite obvious; the lpBytesReturned out argument is only filled if the DeviceIoControl succeeds, and because oSpy’s automatic marshaling code used for logging takes the value of this argument as the size of the ByteArray pointed to by lpOutBuffer, it will go horribly wrong if the app doesn’t initialize the value pointed to by lpBytesReturned to zero.

So, guess this means we need some way to say “don’t log this unless the return value is non-zero”. What I came up with is the following:

So the log condition should be fairly self-explanatory; reg is short for CPU register, eax is the name of the x86 CPU register where the return value is placed with the stdcall calling convention, and then there’s the operator and the value to compare against.

The current implementation is really simple and only supports comparing CPU registers for equality and inequality against a value on the right hand side, but this can easily be expanded. What it does is simply allocating a chunk of memory, filling it with the machine code of a function that gets a pointer to a struct with the CPU registers and returns true/false to determine if the argument should be logged. The chunk of memory is then made executable, a function pointer is pointed to it, and oSpy’s logging code just calls it if it’s set.

The machine code is created from a template with the bytes in the left column:

So the byte in orange is filled in depending on the register, the 32 bit value in blue gets filled with the value to compare against, and the byte in green is set depending on the operator, 0x94 for ==, 0x95 for !=, 0x96 for <=, etc.

This log condition feature was something that I remember thinking about doing and that I ended up putting on hold for a few weeks. One late evening I started thinking about it again and suddenly this idea struck me, so I jumped straight in and implemented it. The really scary part is that it actually worked on the first run. Surreal…


~ by oleandre on June 23, 2008.

13 Responses to “oSpy and JIT x86 machine code generation”

  1. Any hints on getting an oSpyAgent with config.xml support running?

  2. That’s a good question, and I’m glad you asked. 🙂 I’ll try to release a binary snapshot of HEAD so that curious developers can play with oSpy v2. The idea is that it’ll be more like an open invitation for people to get involved than useful at the same level as oSpy v1, namely because there’s not yet any usable UI for visualizing the capture.

    The recommended way of playing with it right now is doing:
    svn checkout http://ospy.googlecode.com/svn/trunk/
    Then use MSVS2005 to open oSpy.sln and build InterceptPP, oSpyAgent and oSpy. (oSpyUsbAgent and oSpyStudio can be ignored for now.) Run oSpy and do a capture, optionally customizing config.xml to suit your needs.

  3. I am having problems hooking the functions CreateThread, HeapAlloc and HeapFree. Like you recommended in your previous posting i recompiled from svn and added some other hooks to the config.xml and it was all working fine. The problems started when i added the hook for the function “CreateThread”. Shortly after injection and running the application seems run into a deadlock (about 20 active threads usually). Now this could be a bug in my application which didn’t show up before.
    But when i tried to use the hooks for HeapAlloc and HeapFree my application deadlocks/freezes immediately when the capture popup appears which itself shows about 60 events and 0 bytes received. The application i study is idle when i inject your agent.

    Any advice?

  4. Sorry there seems to be a problem with embedded xml. Thats relevant section of my config.xml: http://nopaste.ns-linux.org/?M2QxM2

  5. Could you please try to update to SVN HEAD and try again? I’ve made a few improvements that should hopefully fix the re-entrancy issues that you’re experiencing. (If you intercept calls to a function that oSpyAgent itself relies on, things went horribly wrong. 🙂 )

  6. You probably also wanna comment out the body of FunctionCall::AppendBacktraceToElement in InterceptPP’s Core.cpp if the process you’re going to monitor does lots of calls to the functions you’re monitoring, as the backtrace generation code is currently quite slow.

  7. Seems to be working now. Thanks a lot! 🙂

    I have some more questions:
    1. how can i export all the captured events in xml format?
    This could be useful if you want to analyze the captured data using external programs. Simple examples would be to search for a specific event or compare multiple “recordings” of the same program.

    2. is it difficult to implement the hex=”true” for the types UInt32Ptr and ByteArrayPtr?
    Sometimes the hooked functions refer to a big portion of data which can’t even be displayed in the xml output window. At least i seem to be unable to scroll that far to the right. And even if i could i would have to base64decode it every time to see the contents.

  8. Awesome! 🙂 It’s probably a good idea to keep updating to HEAD, I’ve been doing quite a few bug-fixes since my last comment here, and I’m hoping for more improvements to come in the next few days. 🙂

    1) The .osd file format is quite simple, so this could easily be added. For now you can do it manually, just save to an .osd, rename it to .bz2 and decompress with any tool supporting the BZip format (WinRAR for example). What you get then is a file with a 32 bit unsigned int as the first 4 bytes (it contains the number of events in the file for progress-bar purposes), then a BOM (Byte Order Marker) and just plain UTF-8 encoded XML after that.

    2) This would certainly be possible and not very hard, but the idea was initially that a separate UI would be used for the actual analysis/visualization. Ideally this UI would be truly cross-platform. oSpyStudio was such an attempt, but it hasn’t been worked on in a while. I’m tempted to give up on the whole cross-platform idea though and just cram it all into the main UI like oSpy v1, and maybe also use WPF for visualizing things more easily. Feel free to dive in and help out if you like, just drop me an e-mail (oleavr at gmail dot com). (Right now I’m the only one working on the project so things are progressing much slower than what I’d like.)

  9. Hi Oleandre,

    Is this code still maintained? I haven’t seen any updates for the past months, and couldn’t find the source code for v2 in google code…
    Excellent project man!

    • Hi Dan,

      The code is still maintained, although I haven’t had time to work on it in a while. The source code can be found in SVN at:
      There’s no useful UI for v2 yet, but I’m considering reviving the old 1.9.6 UI and make it use the new flexible backend (Intercept++), as I realize I won’t have time to write a cross-platform UI for v2 in some time. I started hacking towards this, but it’s still in the early stage of development (named ‘oSpyClassic’ in trunk). Contributions are most welcome though. 🙂

  10. Hi

    Please contact my at info@joebox.org.

    Kind regards


  11. […] http://ospy.googlecode.com/files/oSpy-1.10.4.zip https://oleandre.wordpress.com/2008/06/23/ospy-and-jit-x86-machine-code-generation/ […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: