Jump to content


Photo

CTD at 3.1GB


  • Please log in to reply
471 replies to this topic

#46 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 07:42 AM

I only have a minute, but I wanted to clear something up before this thread gets too off track. In reference to System Memory (not VRAM): Memory is only paged out to disk if the entirety of physical memory is used up. This is not managed by individual processes, thus this cannot be a limitation of the game engine and is not related to the amount of memory given to a process. To the process, all it sees is the virtual memory addresses. When the process wants to get something from memory, it tells the OS it wants to get (virtual) address, lets say, 57. The OS then translates that into either the physical memory location or the page location on disk. Now on to some of the comments: Disabling paging is a bad idea. It forces, as Salvador points out, all of the memory to be in physical memory and not paged to disk. Unfortunately, if the system runs out of physical memory it simply crashes. No ifs, ands or buts. It won't speed up a program because the OS already is trying to keep everything in memory that it possibly can. On a more technical note, some OSes may page things out to disk before it is necessary; however it keeps it in memory too. This just allows the OS to not have to copy a page from memory to disk in an "emergency". I don't know as much about VRAM usage. From z's post (sorry! I forgot the number's that follow!) it seems that when VRAM runs out, the system allocates some graphics memory in RAM as well. This will increase stuttering, but not cause CTDs. This is described in much more detail in z's performance testing. If the system got to the point where both VRAM and Memory were paged to disk...you're in trouble. If somehow the system had to allocate some page space on disk for VRAM, your game would be so slow it would be unplayable. Don't do it. Buy more ram. Salvador, you mention that this might be a Bethesda issue. That is currently one of the two likely possibilities, and the one we had agreed upon until z came along and suggested the possibility of a setup issue. Not bashing on z, I appreciate help from anyone who can give it. Salvador, you also mentioned the issue of DLLs. You are right, DLLs are allocated in the same address space so they will take up some of the memory. I have found, though I hadn't previously mentioned, that removing ENB does reduce memory usage, but only slightly. I still have the same 3.1GB limit in that case. Also, I don't think our STEP installments are all that similar, though no one else has posted their mod list. Mine, in particular, is not really STEP for more than textures and fixes. In particular, I have several very RAM heavy mods that are not in STEP (nor do I recommend that they be) and it is unlikely that others here have them all installed. Neo, based on his incredible Skyrim Revisited page, has somewhat similar install to mine, but again varies in some very large mods (like Interesting NPCs). I really do not mean to discourage discussion, I just wanted to make sure things stayed on track. I'm not an expert on this topic, by any means, and I am just as baffled by the 3.1GB cap as everyone else. I apologize in advance if I have been rude to anyone...I haven't had my coffee yet...
  • 0

#47 Besidilo

Besidilo

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 993 posts

Posted 27 February 2013 - 07:42 AM

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory. Google "Large Address Aware" if you don't believe me. EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim... https://www.gamasutr...nd_.php?print=1 https://www.techpowe...ead.php?t=91260
  • 0

#48 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 08:51 AM

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory.

Google "Large Address Aware" if you don't believe me.

EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim...

https://www.gamasutr...nd_.php?print=1

https://www.techpowe...ead.php?t=91260

Really,

That is very interesting indeed.... but also worrisome. Because quite a bit of the VRAM is mirrored in RAM. This would suggest that DDSopting would have double the effect?... Unless Z is absolutely right and there is no mirroring going on. Nonetheless, DDSopt seems to be really important now.

@ Viking Yes my bad, it should only page when running out of RAM, however, OS dumps quite some things in Page (that are not needed soon) to free up RAM. Processes can however address page file directly. Still, the process should be able to address 4GB of space with a 32 bit integer.

Besidilo's post would suggest that the other 1GB is in VRAM. Isn't VRAM usage addressed by the graphics driver? 
  • 0

#49 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 12:31 PM

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case.  I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.
  • 0

#50 Besidilo

Besidilo

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 993 posts

Posted 27 February 2013 - 12:52 PM

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.
  • 0

#51 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 27 February 2013 - 01:03 PM

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I have not been formally recording the results, but since I returned to Skyrim a few weeks ago I've been monitoring it's behaviour.  In my case there is no doubt - adding textures does increase both VRAM & RAM usage in a very close ratio.  At the moment, my Skyrim setup is in bits, but I will do some proper benchmarks at a later date.

As Viking is at pains to point out, there is something significant going on here, but so far we know very little.  One fundamental point is that correlation does not prove causation - although the crashes appear to occur at a certain level of memory utilisation, it is possible that other less visible demands increase in tandem with memory usage, and these could be implicated. 

That said, I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.
 
  • 0

#52 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 01:32 PM

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory.

Google "Large Address Aware" if you don't believe me.

EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim...

https://www.gamasutr...nd_.php?print=1

https://www.techpowe...ead.php?t=91260

Alright. I finally had a chance to read through both of these. Up front, the gamasutra article is quite misleading and I wouldn't recommend it. The links it references are all high quality and the author did know what he was talking about (for the most part), but leaves out some essential bits if you aren't familiar with the area. The second link is accurate, but somewhat dated. I don't really feel like going into more detail here, but can if someone is interested.

In response to your comment, we still are not sure exactly how Skyrim manages its vram. A link earlier in this thread (and actually in your first link) discusses the windows patch that helps manage vram outside of the address space of the program. In versions of directx earlier than 10, the vram is mirrored in the application address space, but I haven't seen any clear indications of skyrim being strictly directx 10 or strictly directx 9 (please link if you can), and so, in my opinion, we are unable to answer with any certainty if all vram is always mirrored in application ram for skyrim. 

Note that this is a different issue than if vram on the graphics card is in the same address space as the application. Based on my experience, this is not the case. My skyrim ram + skyrim vram often exceed 4gb.

As for if you should get a 4GB graphics card, I often go above 2GB of VRAM, but have yet to hit 3GB vram. I think you will still find an advantage of having a 3GB+ graphics card.
  • 0

#53 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 01:37 PM

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.

It is a shame that we keep posting at the same time. I assume that your second paragraph means that virtual memory is different than physical memory, and in that we are in agreement.

I'm not sure why you think that you can't measure VRAM usage accurately. I agree it is difficult to pin VRAM usage to a particular application, but you can get pretty good idea.

I addressed your other points in my previous post.
  • 0

#54 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 01:40 PM

That said I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.

I think that this is a very important point. Assuming memory is the issue (which can't do with complete certainty), the issue very easily could be a limitation of the CreationKit engine (is that what it's called?).


  • 0

#55 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 01:44 PM

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.

DX9 is partially fixed, and not fully mirrored anymore, but the allocated VRAM is (the part allocated to Skyrim (textures)) according to besidilo site link (which is really good, love the references). In short, for DX9 the allocated VRAM grows (and is duplicated in RAM) when more VRAM is needed by the game engine and thus more memory is used when more VRAM is allocated (compressed textures I presume).

My understanding from your words, and from quickly reading through the articles is that you should also count the VRAM, at least the part of textures that is loading into VRAM. Not all the AA and AF and other processing going on in your graphics card that is also loaded into VRAM. So, from those 2.7GB only, lets say (just random example), a third are actually loaded textures, the remainder is used and allocated by the driver. However, I was under the impression that the driver handles VRAM allocation completely, which apparently is incorrect, or perhaps the driver sort of functions as the OS does for RAM when it comes to allocating VRAM for Skyrim.

I know only limited tests are done. I never said everything or nothing is mirrored. In the past things were completely mirrored as can be understood from the literature such as besidilo points to. And now this is mostly fixed, apart from DX9 which is only partially fixed. However this does not explain how it comes you can continue to play when VRAM cap is reached. This would suggest that dynamic or system VRAM, is RAM allocated as being VRAM? If that is indeed the case, than that dynamic VRAM (if allocated to Skyrim engine and not the VGA driver) should also be duplicated. This would suggest that above VRAM cap your memory usage grows rather fast. Would be nice to see full RAM usage on Zs benchmarks. You'd expect that an increase in system VRAM of 200mb means a increase in memory usage of 400mb, could perhaps be a nice test to measure if there is mirroring going on.

Love this thread and these type of discussions. Apologies to viking for changing the topic quite a bit.
  • 0

#56 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 01:49 PM

I see I'm slow with posting. @Viking As I liked the gamasutra link, due to the good references, I would love to know what is so misleading about it. It did indeed mis the info in the earlier link that showed that for DX9 only allocated VRAM is mirrored and for DX10 and 11 nothing is mirrored in RAM anymore (after fix). I believed Skyrim was DX9, but apparently it could be DX10 as well? both XBOX and playstation are DX9 only right? Or is PC version considerably different?
  • 0

#57 Besidilo

Besidilo

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 993 posts

Posted 27 February 2013 - 01:50 PM

VRAM usage is almost never fixed. It's dynamic + memory buffer. Which is why it's pretty much impossible to measure it easily or accurately for any game. No, MSI Afterburner or GPU-Z won't do the job, as they will only show you the allocated memory, with no regards as to what is actually being buffered to the memory, how much is used by the application, how much by background process or system UI. Oh, and it's Besi·dilo. And no, Skyrim isn't "partially DirectX 9". It's fully DirectX 9 and there are no Dx10/11 executables. So the normal memory limitations apply.
  • 0

#58 Besidilo

Besidilo

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 993 posts

Posted 27 February 2013 - 01:55 PM

 

That said I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.

I think that this is a very important point. Assuming memory is the issue (which can't do with complete certainty), the issue very easily could be a limitation of the CreationKit engine (is that what it's called?).

 


The engine is called Creation Engine.

https://elderscrolls...Creation_Engine
https://www.rockpape...ngine-detailed/


  • 0

#59 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 02:01 PM

VRAM usage is almost never fixed. It's dynamic + memory buffer.

Which is why it's pretty much impossible to measure it easily or accurately for any game. No, MSI Afterburner or GPU-Z won't do the job, as they will only show you the allocated memory, with no regards as to what is actually being buffered to the memory, how much is used by the application, how much by background process or system UI.

Oh, and it's Besi·dilo.

And no, Skyrim isn't "partially DirectX 9". It's fully DirectX 9 and there are no Dx10/11 executables. So the normal memory limitations apply.

Hmmm, how did I got your name wrong?.. apologies.

I believed it was DX9 only, yet Viking said it could partially be DX10. If DX9 only this means that all allocated VRAM to skyrim is mirrored in system memory.
Yes I know it is all allocated and not necessarily used. But this should suggest, that any increase in dynamic VRAM (or system VRAM) is mirrored in RAM. Therefore, by just increasing texture sizes when you reached your VRAM cap, you could perhaps get an idea on how much is really mirrored (allocated. The problem occurs with 3.1GB allocated memory).

However, I need to find the link, in original windows 7 RAM allocated by sytem was shown, and because of the improved pre-fetch, it had very high memory usage numbers, which upset quite a lot of people, so they changed that. So even the allocation numbers are somewhat incorrect.

Edit: fixed names
  • 0

#60 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,623 posts

Posted 27 February 2013 - 02:18 PM

Looks like I sparked up quite a discussion! :D

It only popped up in my head last night... "hasn't any thought of the RAM limitation with x86 architecture?"

It just made since in my mind that any programs that passes this limit and was build on this architecture is going to have issues. Since Skyrim was build essentially with Windows XP era technology (DX9, etc) then this limitation should also apply to Skyrim. Keeping that logic in mind, if Skyrim processes are trying to allocation more than 4GB total memory, you could expect issues to arise. (CTD, freezing, etc).

EDIT:
Who marked this thread as solved because the discussion is clearly continuing and unclear of what is causing the real issue. I'm taking it off.

EDIT: I know see Viking did it in the title.
@ Viking,
The admins or moderators (myself) will mark the thread as solved when the issues have clearly be resolved. We have a special "pre-tag?" for this so there is no need to edit the title to mimic it.


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users