I set the customer as non-taxable but still when I create a new invoice it adds the tax which I have to remember to take off each time?

Hello there, kalvinb76.

Aside from ensuring that your customer profile is tax-exempt, you'll also have to set your products and services as non-taxable in their category and manually uncheck the Tax column box when creating an invoice. 

To set up non-taxable category items:

1.  Go to the Gear icon at the top.
2. Select Products and services.
3. Edit the item in question and ensure that the Sales tax category drop-down is Non-taxable. 
4. Click Save.

Right after changing the sales tax category, untick the Tax column on the invoice. Please see attached screenshot:

That's it! You can also, check out this article to learn  on how QuickBooks calculates sales tax for reference.

Stay in touch on how this goes by commenting below. I'm always around whenever you need anything else.