Windows normally allows user mode applications to use 2GB of memory. The switch moves the starting point of the kernel to 3GB making it available to user mode applications. Windows 2003 added an extra switch userva to customize the exact amount between 2GB and 3GB. In short this makes it possible for your applications to use 3GB instead of 2GB of memory. Do note that there may be some drawbacks when your server is very powerful. In some cases the server might be able to handle so many threads that it's 1GB of kernel memory is insufficient.
Note Microsoft Product Support Services strongly recommends using a range
of memory for the /USERVA switch that lies within the range of 2800-3030. This
range is wide enough to provide a large enough pool of system page table entries
for all currently observed issues. Usually a setting of /userva=2800 will
provide close to the maximum available number of system page table entries
PAE (Physical Address Extension)
PAE allows Windows to use more memory than 4GB in combination with AWE
AWE is an API that enables programs to address more than 4GB (with PAE). It enables them to reserve memory as 'non-paged' memory. This means that memory taken by AWE is never paged to disk until the applications is closed or explicitly frees it.