Ensure Photoshop 2017 is closed when using regedit.
Run regedit, go to: hkey current user > software > adobe > photoshop >
Right click in right pane > new dword 32 bit value
Type > OverridePhysicalMemoryMB
Open it > change to decimal > enter 8000 (eg 8000 MB for 8 GB RAM)
Click ok and close regedit