QuickBooks Online allows you to enter negative amounts. However, you must not have a negative Total or BALANCE DUE on the invoice. Otherwise, an error like you have mentioned will occur.
Instead, we can create a Credit Memo. This will set a negative balance on the customer's account. You can also apply this to future invoices of the customer. You can use this for reference: Create and apply credit memos or delayed credits in QuickBooks Online.
Here's also an article that is related to credit memos: Handle a customer credit or overpayment in QuickBooks Online.
Feel free to get back to us if you have additional queries. Have a great day!