Using Autodafe is described in "Gray Hat Hacking, Third Edition" (recently just came out!) under the SCADA hacking section.
There is also a great guide that discusses how to use The Peach Fuzzing Frameworks' peachshark along with Wireshark in the book "Hacking Exposed Windows, Third Edition".
If the protocol is MSRPC, it may be a little more difficult, but there are tools out there such as this one: http://wiki.austinhackers.org/2006-11-29-0x0003
You may want to try the ProxyFuzz script included with taof before anything else, as it if finds something -- then you are golden right away with almost no configuration or advanced fuzzing. If not, then you might want to move to reversing the application on both sides (client and server) and establishing your own rules about their protocol use. EFS (as shown in the last chapter of "Open-Source Fuzzing Tools") is a great way to utilize PIDA (Python IDA) files to do a lot of this work for you, as is CatchConv (a Valgrind plugin), also described in that chapter.
Ideally, you would want to understand both the runtime and static (i.e. deadlist) views of the target apps including their infrastructure. Different tools call for different measures. You can dump your DRAM. You can hook system or library calls that access the network. You can locally proxy a network connection. You can attach debuggers, software/system-call/library-call tracers, or fault-monitors of various types. You can run it through emulation, or inside a virtual machine. You can watch CPU registers. You can trace instructions or functions. You can implement dynamic binary instrumentation to insert your own code into the code, in order to understand the baseline code better.
Some of this is dependent on the level of indirection you're working with. Is it managed code? Does it implement its own virtual machine or p-code? Is it a PE or ELF file? Can you identify how the program is installed, and what files and registry settings it affects when installed? Does it install any protocol handlers (ViewPlgs.exe may help)? Does it register or call any services? What libraries does it depend on? Does it contain any strings in the binary that appear to be banned functions (you can do this using BinScope under Windows)? Do you have debugging info, or is the binary stripped? Do you have a symbol table, or can you identify them or download/import them? Does the code contain any easily identifiable or interesting components (you can use signsrch to help with this)? Can you inject shared libraries and what is their affect? Is the binary or its source code using any self-modifying or self-checking code (again, signsrch may help here)? Is there an underlying protocol with a known specification (e.g. IETF RFC) available, or can it be elicited or reverse engineered easily?
If you've come this far with a few conclusions (but no results), you will need to design a methodology for code audting. I suggest checking out chapter 4 of "The Art of Software Security Assessment" in order to learn about external flow sensitivity and tracing direction. You'll probably want to consume all of the information available here -- http://pentest.cryptocity.net/reverse-engineering/
