It's entirely possible that VMware Fusion would generate inactive memory (particularly through disk buffering, as mentioned above by Linjo), but it would seem that in your case the OS is failing to convert the inactive memory back to free memory at the appropriate time – i.e. when the system is low on free RAM. The subsequent lack of free memory would be expected to lead to the performance issue you describe.
Perhaps VMware Fusion is the only program you are running which has a memory (and/or disk) workload likely to lead to the ongoing generation of inactive memory, and is thus the only program which triggers the problem?
Cheers,
--
Darius