What do you do with a open balance transaction in the vendor Information view?
You should have a negative open balance if you have a credit with the vendor. If it's a positive balance, then you have an outstanding balance you owe. It sounds like you have balances when you don't owe or are owed anything, yes?
When you wrote the check to the vendor that wasn't for a bill, what was it for?
Credit card charges should be entered as credit card charges (Banking>Enter Credit Card Charges) and assigned to the appropriate expense or asset account. Those charges should not impact your vendor balances unless you're choosing account payable as the expense account.