Someone please explain this RAM limitation to me.

[NecroBones]

4 minutes ago, godefroi said:

A 32-bit program can use more than 2GB, 3GB, or even 4GB of memory, quite easily. The limit that applies to 32-bit software is ADDRESS SPACE, not memory. There's ways to use more memory than you have address space for, but Squad doesn't use them. Even then, there's techniques (such as unloading bits of data that you're not using/won't use soon) that could be applied, but aren't. Regardless, the problem will (mostly) go away in 1.1. Until then, use Linux.

 

I'm having flashbacks to the days of HIMEM.SYS and EMM386.

 

[Foxster]

31 minutes ago, NecroBones said:

 

I'm having flashbacks to the days of HIMEM.SYS and EMM386.

 

If that gets your memory juices flowing then how I spent today with 15 students working on 35 year old BBC Micros would have really had you going!

[NathanKell]

19 hours ago, Foxster said:

If that gets your memory juices flowing then how I spent today with 15 students working on 35 year old BBC Micros would have really had you going!

Oh myyyy

[Renegrade]

On 2015-12-04, 8:02:18, Foxster said:

If that gets your memory juices flowing then how I spent today with 15 students working on 35 year old BBC Micros would have really had you going!

Nice!  I gotta get off my lazy butt and source a replacement video chip for that C64 (C-model) I have laying around...  The 6502 era was a really interesting time.

[jwbrase]

On Friday December 04, 18:23:52 GMT-0600, godefroi said:

The sheer amount of misunderstanding/misinformation/straight-up lies in this thread is staggering.

A 32-bit program can use more than 2GB, 3GB, or even 4GB of memory, quite easily. The limit that applies to 32-bit software is ADDRESS SPACE, not memory. There's ways to use more memory than you have address space for, but Squad doesn't use them. Even then, there's techniques (such as unloading bits of data that you're not using/won't use soon) that could be applied, but aren't. Regardless, the problem will (mostly) go away in 1.1. Until then, use Linux.

 

The use of mechanisms such as overlays is rare enough in consumer software on modern operating systems that you will rarely, if ever, these days encounter a program that uses more physical memory than it has address space available to it. And if your program is meant to be cross platform, like KSP, the "quite easily" part goes away: Pretty much every way I'm aware of to do overlays, etc. is system-specific.

[Daid]

On 5-12-2015 01:23:52, godefroi said:

The sheer amount of misunderstanding/misinformation/straight-up lies in this thread is staggering.

A 32-bit program can use more than 2GB, 3GB, or even 4GB of memory, quite easily. The limit that applies to 32-bit software is ADDRESS SPACE, not memory. There's ways to use more memory than you have address space for, but Squad doesn't use them. Even then, there's techniques (such as unloading bits of data that you're not using/won't use soon) that could be applied, but aren't. Regardless, the problem will (mostly) go away in 1.1. Until then, use Linux.

First bit. Yes. There is a lot of misinformation about memory and the 2/3/4GB limit.

But I'm not sure how a program can use more then the 4GB limit. If you cannot address the memory, how can you use it? Sure, the kernel can do all kinds of things, with PAE the 32bit address space for even a 32bit OS is not a real issue. But your application has no use for memory it cannot access. So I would love to know how you can access more then that?

(The 2GB limit is just an very old limit in Windows. It's 1 compile flag to enable it up to 3 or 4GB depending on other factors, so usually nobody talks about this limit anymore)

[oglommi]

On 5.12.2015, 01.29.39, NecroBones said:

 

I'm having flashbacks to the days of HIMEM.SYS and EMM386.

 

Qemm386 ruled

Edited December 7, 2015 by oglommi

[Curveball Anders]

On 2015-12-06 11:11:37, jwbrase said:

And if your program is meant to be cross platform, like KSP, the "quite easily" part goes away: Pretty much every way I'm aware of to do overlays, etc. is system-specific.

Even more so if your program not only is cross platform, but also developed in an existing external engine like Unity.

[jwbrase]

18 hours ago, Daid said:

But I'm not sure how a program can use more then the 4GB limit. If you cannot address the memory, how can you use it? Sure, the kernel can do all kinds of things, with PAE the 32bit address space for even a 32bit OS is not a real issue. But your application has no use for memory it cannot access. So I would love to know how you can access more then that?

A common way of doing it is to have a certain region of the address space that can be remapped to different areas in memory, either by hardware bank switching (which was common on old 8 and 16 bit systems, where the technique was most prevalent), or by calls to the operating system that cause it to remap that region of the process's address space on modern systems with paged virtual memory (but it doesn't tend to be that common on modern systems because most programs tend to have lots more address space than they need). The specifics differ between operating systems, and between different hardware on the old 8/16 bit systems. Windows, IIRC, has system calls explicitly meant to provide such overlays. On Linux it would probably be done by switching out between different anonymous mappings to the same area of the address space using mmap().
 

[steve_v]

*recalls fighting for real-mode memory, VGA memory holes and BIOS remapping*
*shudders*

Breaking the 32bit address space limit is non-trivial, and usually platform specific... Unity doesn't try, so we're stuck with it.

19 hours ago, oglommi said:

Qemm386 ruled

JEMM386 is better

[godefroi]

On 12/7/2015, 7:43:28, Daid said:

But I'm not sure how a program can use more then the 4GB limit. If you cannot address the memory, how can you use it? Sure, the kernel can do all kinds of things, with PAE the 32bit address space for even a 32bit OS is not a real issue. But your application has no use for memory it cannot access. So I would love to know how you can access more then that?

First of all, "out of memory" nowadays is never about memory. It's about the address space. You don't run out of memory, because Windows (or whatever OS you use) will happily use swap space to fill in for any physical RAM you're lacking. It's not going to be fast, but "out of memory" you will not be. You run into problems when you run out of address space.

One of the easy ways to use more memory than you have address space is to map a file onto virtual memory. You can make the "file" as big as you want, but you can only see parts of it that you have enough address space for. Fortunately, you can move your "window" around the "file" to see any part you want at any time. This is done with CreateFileMapping() and MapViewOfFile() in Windows, and mmap() in Linux (using MAP_ANONYMOUS, I believe).

Also a common but incorrect belief is that PAE expands the address space of a process. It doesn't. Remember, it's PHYSICAL address extensions, meaning it lets the processor address more than 4GB. 32-bit programs are still limited to the 4GB address space (of which, 2GB or 1GB is consumed by the operating system). Even Address Windowing Extensions doesn't do that. 32-bit pointers are 32 bits log, and that's as long as they'll ever be.

Finally, there's no reason that a Unity program couldn't use one of these techniques. Doing so would definitely tie you to a specific platform, but that could be handled using conditional compilation and distributing separate binaries.

I am 100% not advocating that Squad do any of this, but it is incorrect to say that it wouldn't be possible to do it.

Edited December 11, 2015 by godefroi

[magnemoe]

On 8.12.2015, 10.38.54, jwbrase said:

A common way of doing it is to have a certain region of the address space that can be remapped to different areas in memory, either by hardware bank switching (which was common on old 8 and 16 bit systems, where the technique was most prevalent), or by calls to the operating system that cause it to remap that region of the process's address space on modern systems with paged virtual memory (but it doesn't tend to be that common on modern systems because most programs tend to have lots more address space than they need). The specifics differ between operating systems, and between different hardware on the old 8/16 bit systems. Windows, IIRC, has system calls explicitly meant to provide such overlays. On Linux it would probably be done by switching out between different anonymous mappings to the same area of the address space using mmap().
 

This is true, the 8086 and 286 cpu was 16 but but used an 20 and 24 bit address space. the 8 bit cpu had an 16  bit adress space. 
32 bit and 64 bit cpu with modern OS also uses memory mapping to protect memory so you can not overwrite the memory space for other processes, this was an common issue in dos and 16 bit windows. 

However the 4GB limit was not an practical problem outside of server use before 64 bit cpu was common.
Core 2 was Intel first consumer CPU and was released in 2006, AMD was earlier with 64 bit. 
Windows server 2003 was the first microsoft os with 64 bit, its core was also used for an 64 bit version of XP.
Windows vista was not very popular so most continued to run 32 bit xp even if they had 64 bit cpu.

Most programs also don't use anything close to 2GB so you can just run them as 32 bit. 
Most games until the last years was designed to run on consoles with 512 MB ram so they hardly needed 64 bit.
This has changed, games for PS4, One and PC tend to be 64 bit, lots of professional memory hungry software is 64 bit only. 

[Renegrade]

9 hours ago, godefroi said:

First of all, "out of memory" nowadays is never about memory. It's about the address space. You don't run out of memory, because Windows (or whatever OS you use) will happily use swap space to fill in for any physical RAM you're lacking. It's not going to be fast, but "out of memory" you will not be. You run into problems when you run out of address space.

Er, the way you wrote that suggests that swap is unlimited in size *cough*.   Hard drives, even the big scary bit-devouring magnetic beasts, do have limits. (otherwise true though)

 

Quote

Also a common but incorrect belief is that PAE expands the address space of a process. It doesn't. Remember, it's PHYSICAL address extensions, meaning it lets the processor address more than 4GB. 32-bit programs are still limited to the 4GB address space (of which, 2GB or 1GB is consumed by the operating system). Even Address Windowing Extensions doesn't do that. 32-bit pointers are 32 bits log, and that's as long as they'll ever be.

This is correct, although I seem to recall that enabling PAE at kernel compile time lets you drastically shrink the OS footprint in your virtual space (down from 1GiB to a handful of megabytes) in Linux.

[NeMeSiS]

On 6-12-2015 11:11:37, jwbrase said:

 

The use of mechanisms such as overlays is rare enough in consumer software on modern operating systems that you will rarely, if ever, these days encounter a program that uses more physical memory than it has address space available to it. And if your program is meant to be cross platform, like KSP, the "quite easily" part goes away: Pretty much every way I'm aware of to do overlays, etc. is system-specific.

Ah yes, i remember Operation Flashpoint getting the '-nomap' parameter in an update somewhere in 2002 or 2003, for pretty much the exact same reasons as KSP need(ed) it. Too may mods pushing the game over its limit. It seems squad had different priorities the last few years, though it doesn't matter at this point with a stable 64bit version coming out in the next few months.

Edited December 11, 2015 by NeMeSiS

[Kerbart]

10 hours ago, godefroi said:

Finally, there's no reason that a Unity program couldn't use one of these techniques. Doing so would definitely tie you to a specific platform, but that could be handled using conditional compilation and distributing separate binaries.

I am 100% not advocating that Squad do any of this, but it is incorrect to say that it wouldn't be possible to do it.

Let's not forget that there are two memory hogs in the game:

• Textures for parts
• Memory leaks (which seem to be accelerated by some mods)

Having some convulated roll-out-your-own EMS/XMS driver seems a bit of overkill for implementing on-demand texture loading, solving the first problem. As for the second problem... no amount of memory will ever solve that.

[radonek]

11 hours ago, godefroi said:

…Finally, there's no reason that a Unity program couldn't use one of these techniques.…

I can think of few. Can GPU work with remapped memory? Can you do DMA from remapped region? Can drivers handle it?  Without affecting performance? How are other libraries affected?  What about increased TLB pressure? Is it coincidence that database people, who are probably  only one actually using PAE, invent weird stuff like hugetlbfs, do their own disk io etc? And how could any of these be solved when windows coders, as exhibited right here, could not be bothered even with a proper architecture port?

2 hours ago, magnemoe said:

Most programs also don't use anything close to 2GB so you can just run them as 32 bit. 

This is just excuse  of lazy windows coders. 32bit code on 64bit os  have to through emulation layer.  64bit system is not just more bits in cpu registers, its different hardware architecture. And observe - linux community took it as such, did a full port and use 64bit systems for years without problems. As exhibited by Unity. Windows world employ this half-assed approach and ended with half-assed results, as also visibly exhibited by Unity.

1 hour ago, Kerbart said:

Having some convulated roll-out-your-own EMS/XMS driver seems a bit of overkill for implementing on-demand texture loading, solving the first problem. As for the second problem... no amount of memory will ever solve that.

Amen.

[Guest Space Cowboy]

Come on - 386MAX was tha king

[godefroi]

 

1 hour ago, radonek said:

I can think of few. Can GPU work with remapped memory? Can you do DMA from remapped region? Can drivers handle it?  Without affecting performance? How are other libraries affected?  What about increased TLB pressure? Is it coincidence that database people, who are probably  only one actually using PAE, invent weird stuff like hugetlbfs, do their own disk io etc? And how could any of these be solved when windows coders, as exhibited right here, could not be bothered even with a proper architecture port?

None of that is a reason why it wouldn't work. Yes, it'd be slower, but as we like to say, "slow" is still infinitely faster than "crashed".

Also remember, none of this relates to PAE in any way. If you have a 64-bit CPU, PAE does nothing for you. At all. PAE is a CPU thing, not a software thing. Doesn't change how much RAM your program can access, doesn't change address space size. Only lets your 32-bit machine have more than 4GB physical RAM.

[Stone Blue]

On 12/3/2015, 8:22:40, Dafni said:

Hmm, good point. Now you have me thinking. I was an Ubuntu user for a while and really loved it. Now I got that laptop with Win8.1 pre-installed and I thought well, for as long as it runs smooth I keep it. But this sounds actually very tempting now. Thank you for this information.

@Dafni If you are already familiar with Ubuntu, and you mainly JUST want to play KSP, I also recommend a dual-boot with Linux...(Or, put your install on a flash drive)...

I triple-boot my laptop with Win7/8.1 & Lubuntu... I chose Lubuntu for the simple desktop...It only uses 250MB of RAM at idle...ALL the rest of my RAM is available for KSP...
I rarely get crashes... I do sometimes stop & restart the game due to slowdowns & lag, though... i think its mainly related to heat buildup in my laptop tho...I know when my comp completely shuts down, its due to the heat, anyway... lol

Edited December 11, 2015 by Stone Blue

[oglommi]

On 8.12.2015, 12.03.03, steve_v said:

*recalls fighting for real-mode memory, VGA memory holes and BIOS remapping*
*shudders*

Breaking the 32bit address space limit is non-trivial, and usually platform specific... Unity doesn't try, so we're stuck with it.

JEMM386 is better

Never used that one. I recall constant tinkering with the autoexec.bat and config.sys files and several different copies of the files for different games.

[steve_v]

7 hours ago, oglommi said:

Never used that one. I recall constant tinkering with the autoexec.bat and config.sys files and several different copies of the files for different games.

That's because it's new.
It doesn't come with the auto-configuration stuff that Quarterdeck did, but it can move pretty much anything out of that precious lower 640k.
Yes, some crazy fools are still writing stuff for DOS... that gem comes with FreeDOS 1.1.

[jwbrase]

20 hours ago, Renegrade said:

 

 

This is correct, although I seem to recall that enabling PAE at kernel compile time lets you drastically shrink the OS footprint in your virtual space (down from 1GiB to a handful of megabytes) in Linux.

Well, you can compile Linux to put the kernel into its own address space, which frees up most of the top GiB for userspace code (except for the part of the kernel needed to switch address spaces, and probably at least parts of most/all device drivers as well). The disadvantage is that now every system call has to switch address spaces, which has an impact on performance.
 

17 hours ago, radonek said:

32bit code on 64bit os  have to through emulation layer.

Not true on x86. x86_64 CPUs contain back compatibility features that allow 32-bit programs to run unemulated on 64-bit operating systems. The operating system does need to provide both 32-bit and 64-bit versions of its ABI, and drivers need to be 64-bit-specific, but 32 bit application code runs unmodified and unemulated.
 

16 hours ago, godefroi said:

If you have a 64-bit CPU, PAE does nothing for you. At all.

Not strictly true. 64-bit operating systems have to enable PAE as part of the process of entering 64-bit mode at boot, as the 64-bit paging scheme is a variant of the PAE scheme.
 

[radonek]

58 minutes ago, jwbrase said:

x86_64 CPUs contain back compatibility features that allow 32-bit programs to run unemulated on 64-bit operating systems. The operating system does need to provide both 32-bit and 64-bit versions of its ABI, and drivers need to be 64-bit-specific, but 32 bit application code runs unmodified and unemulated.
 

…which is what I call an emulation layer. Small part of it, anyway. Real fun starts with libraries.
 

[Red Iron Crown]

45 minutes ago, radonek said:

…which is what I call an emulation layer. Small part of it, anyway. Real fun starts with libraries.
 

Emulation traditionally means translating the instructions from one set to another for execution (with the associated performance overhead that entails). x86_64 CPUs require no translation to run x86 code (32 or 16-bit), it runs natively on the bare metal.

[magnemoe]

On 11.12.2015, 17.08.24, radonek said:

This is just excuse  of lazy windows coders. 32bit code on 64bit os  have to through emulation layer.  64bit system is not just more bits in cpu registers, its different hardware architecture. And observe - linux community took it as such, did a full port and use 64bit systems for years without problems. As exhibited by Unity. Windows world employ this half-assed approach and ended with half-assed results, as also visibly exhibited by Unity.

Amen.

Someone in another tread described the benefit of using 32 bit programs in linux. 
Still for windows its plenty of 32 bit versions around so it make sense to make small programs with small demands 32 bit.

For some weird reason windows 10 has an 32 bit version. no idea why