At this time, QuickBooks Online (QBO) doesn't have the option to change the format to 3 decimal place. You can see other formats available by going to your Accounts and Settings. I'll show you the steps below:
At the top right, click the Gear icon.
Select Account and Settings.
On the left pane, select Advanced.
Click anywhere inside Other preferences.
Select your preferred Number format on the drop-down.
Being able to widen the customization option for decimal is a good idea. While we continue to work on improving the features of the product, I want you to stay updated with the latest QuickBooks news and updates by visiting our QuickBooks blog.
Feel free to comment below if you have further questions. I would appreciate the opportunity to help you with your concerns.