RE: Can I set up the nonprofit as a new company and properly manage it as a nonprofit with the current software?
Yes, you can create a new company that's a non-profit. There's no need to use the non-profit edition to keep the books for a non-profit company.
RE: If so, what do I need to pay attention to when setting up the company?
I don't think really anything special. There's a short interview that asks you about the company, which should work for a non-profit.