Jump to content


Photo

CTD at 3.1GB


  • Please log in to reply
471 replies to this topic

#61 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 02:22 PM

I'm going to try to get the thread back on track: Besidilo, I'm not sure who got your name wrong. If it was me, sorry about that. I know very little about VRAM. A link with more info would be more helpful. The more we understand about the behavior of VRAM the more likely we are to understand this issue. I read in some forum somewhere (highly reliable...) that Skyrim was partially directx10. Don't trust me on that. It makes far more sense for it to be directx9, especially since xbox is limited to that. Salvador, I agree that the references is Besidilo's links were excellent. I just had a hard time following the first one. I didn't notice anything incorrect in the links, there were just a few times when things like the maximum ram in a system was taken to be a constant across all systems. Again, the information was correct, just difficult to follow and left some key pieces of information out IMO.  z recommended several test we can run to test the memory/vram correlation, and I will run them this weekend. Are there any other ideas on how we can get to the root of this issue? 
  • 0

#62 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 02:24 PM

Sorry, I had changed the thread to solved...because at one point it seemed like we weren't going to get any further. This has obviously changed so thanks to whomever changed it back.
  • 0

#63 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 02:25 PM

VRAM usage is almost never fixed. It's dynamic + memory buffer.

Which is why it's pretty much impossible to measure it easily or accurately for any game. No, MSI Afterburner or GPU-Z won't do the job, as they will only show you the allocated memory, with no regards as to what is actually being buffered to the memory, how much is used by the application, how much by background process or system UI.

Oh, and it's Besi·dilo.

And no, Skyrim isn't "partially DirectX 9". It's fully DirectX 9 and there are no Dx10/11 executables. So the normal memory limitations apply.

Hmmm, how did I got your name wrong?.. apologies.

I believed it was DX9 only, yet Viking said it could partially be DX10. If DX9 only this means that all allocated VRAM to skyrim is mirrored in system memory.
Yes I know it is all allocated and not necessarily used. But this should suggest, that any increase in dynamic VRAM (or system VRAM) is mirrored in RAM. Therefore, by just increasing texture sizes when you reached your VRAM cap, you could perhaps get an idea on how much is really mirrored (allocated. The problem occurs with 3.1GB allocated memory).

However, I need to find the link, in original windows 7 RAM allocated by sytem was shown, and because of the improved pre-fetch, it had very high memory usage numbers, which upset quite a lot of people, so they changed that. So even the allocation numbers are somewhat incorrect.

Edit: fixed names

Since there is an absolute allocation cap at 4GB, and since allocation has improved considerably (read most skyrim updates), especially in combination with ATTK. You can argue that when you get CTD, you have used most of the allocated memory. If not, there is still room for improvement. Usually when the process needs to allocate more, it removes blocks (or pages) it believes it no longer requires. If it fails to do so, it is unable to allocate more.
  • 0

#64 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,623 posts

Posted 27 February 2013 - 02:33 PM

Thread isn't off track from what I've read. The idea I presented was just another possible cause of what is causing the issue and everyone has been discussing it. Z's suggestion is also to measure the VRAM and system RAM using the STEP benchmarks. I think the 4GB limitation could be a very real possibility of the cause of the issue. Now, the only real way to test if this is a RAM limitation issue is to have others heavily mod and use RAM heavy mods to push their systems to and beyond what you have done. If everyone gets a crash between 3 and 3.5GB, then it would be safe to say that we've found the limit to modding Skyrim when it comes to RAM useage.
 

Sorry I had changed the thread to solved...because at one point it seemed like we weren't going to get any further. This has obviously changed so thanks to whomever changed it back.

That was me. We don't mark anything as solved unless it is truly solved. If it remains unclear or unknown what caused the issue, then it will not be marked as solved. Even if it's no longer being discussed.


I'd like to point out, if it hasn't already, for anyone coming into this thread and reading up on it, that this is not a STEP issue as hundreds are running pure STEP without incident.



#65 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 02:39 PM

Thread isn't off track from what I've read. The idea I presented was just another possible cause of what is causing the issue and everyone has been discussing it. Z's suggestion is also to measure the VRAM and system RAM using the STEP benchmarks. I think the 4GB limitation could be a very real possibility of the cause of the issue. Now, the only real way to test if this is a RAM limitation issue is to have others heavily mod and use RAM heavy mods to push their systems to and beyond what you have done. If everyone gets a crash between 3 and 3.5GB, then it would be safe to say that we've found the limit to modding Skyrim when it comes to RAM useage.

I'd like to point out, if it hasn't already, that this isn't a STEP issue as hundreds are running STEP without problems.

I fully agree.

Would be nice to see memory hogs that do not increase textures. That way the mirroring can be tested... The benchmark the guy used for testing ATTK was cheese wedges. These are duplications, and therefore require no additional textures or VRAM presumably. But their havoc needs calculations and thus a lot of memory. Is that an idea?
BTW I couldn't find the link about the prefetch stuff I was talking about, I know it was from Mark Russinovich and this comes close. Although it is no longer shown in the task manager, that is still happening!

EDIT: Perhaps not the best idea. The graphs that were provided by xeightballx show no or little increase in RAM usage (or CPU?), but he already reached his VRAM max. Also VRAM and RAM usage remained high, even though the game had crashed?
  • 0

#66 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 27 February 2013 - 02:46 PM

... we still are not sure exactly how Skyrim manages its vram. A link earlier in this thread (and actually in your first link) discusses the windows patch that helps manage vram outside of the address space of the program. In versions of directx earlier than 10, the vram is mirrored in the application address space, but I haven't seen any clear indications of skyrim being strictly directx 10 or strictly directx 9 (please link if you can), and so, in my opinion, we are unable to answer with any certainty if all vram is always mirrored in application ram for skyrim.

I think this is a key question.  The wording in the KB article (attached to a hotfix), suggests that Windows itself was modified to improve the problematic memory allocation of DX9:

If an application creates its own in-memory copy of its video resources, or the application uses DirectX 9 or an earlier version, the virtual address space contains the WDDM video memory manager's virtualized range and the application's copy. Applications that use graphics APIs that are earlier than DirectX 10 and that target GPUs that have large amounts of video memory can easily exhaust their virtual address space.

To address this problem, Microsoft is changing the way that the video memory manager maintains the content of video memory resources. This change is being made so that a permanent virtual address range does not have to be used for each virtualized allocation. With the new approach, only allocations that are created as "lockable" consume space in the virtual address space of the application. Allocations that are not created as "lockable" do not consume space. This approach significantly reduces the virtual address space that is used. Therefore, the application can run on large video memory configurations without reaching the limits.


This seems to leave open the question of how much of Skyrim's memory texture caching is inherent to its DX9 origins, or a (possibly configurable) feature of the engine.  Perhaps someone knows with certainty the DX9 memory behaviour on later versions of Windows?
  • 0

#67 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 02:59 PM

 

... we still are not sure exactly how Skyrim manages its vram. A link earlier in this thread (and actually in your first link) discusses the windows patch that helps manage vram outside of the address space of the program. In versions of directx earlier than 10' date=' the vram is mirrored in the application address space, but I haven't seen any clear indications of skyrim being strictly directx 10 or strictly directx 9 (please link if you can), and so, in my opinion, we are unable to answer with any certainty if all vram is always mirrored in application ram for skyrim.

I think this is a key question.  The wording in the KB article (attached to a hotfix), suggests that Windows itself was modified to improve the problematic memory allocation of DX9:

If an application creates its own in-memory copy of its video resources, or the application uses DirectX 9 or an earlier version, the virtual address space contains the WDDM video memory manager's virtualized range and the application's copy. Applications that use graphics APIs that are earlier than DirectX 10 and that target GPUs that have large amounts of video memory can easily exhaust their virtual address space.

To address this problem, Microsoft is changing the way that the video memory manager maintains the content of video memory resources. This change is being made so that a permanent virtual address range does not have to be used for each virtualized allocation. With the new approach, only allocations that are created as "lockable" consume space in the virtual address space of the application. Allocations that are not created as "lockable" do not consume space. This approach significantly reduces the virtual address space that is used. Therefore, the application can run on large video memory configurations without reaching the limits.


This seems to leave open the question of how much of Skyrim's memory texture caching is inherent to its DX9 origins, or a (possibly configurable) feature of the engine.  Perhaps someone knows with certainty the DX9 memory behaviour on later versions of Windows?

 

Install skyrim on XP x64 with only DX9 installed and see if it runs. If it does, it certainly doesn't use DX10.

What is defined by lockable? And is all allocated VRAM allocated by Skyrim? Or part (as I assume) by the driver to process the images.... Would it make a difference to run AA through driver or through skyrim? Would this be a separate process? ENB DLL (and therefore I assume AA through Skyrim) takes up memory in the skyrim process for sure. 

EDIT: Says on the box, DirectX 9c video card. So no DX10, although I am sure most people already assumed so. Windows did fix allocation. Now the full 2GB isn't mirrored anymore, but the mirror increases with VRAM demand from the process. But apparently that is dependant on "Lockable" allocation

EDIT EDIT: apologies, VRAM is part of the allocated virtual memory space. And all "lockable" allocations take space within the allocated virtual memory. However, VRAM used by the process is mirrored in RAM for DX9. The question now remains. Is all lockable allocated VRAM mirrored? What is lockable?


  • 0

#68 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 27 February 2013 - 03:09 PM

Install skyrim on XP x64 with only DX9 installed and see if it runs. If it does, it certainly doesn't use DX10.

Just to clarify - I'm not questioning whether Skyrim is DX9/10.  The article seems to suggest that Windows was patched to improve the memory handling of legacy DX9 applications, without updating them to DX10. 
  • 0

#69 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,623 posts

Posted 27 February 2013 - 03:20 PM

The way you're highlighted text reads (and this just might be me), is that it depends on how the DX9 application (game) is written to handle the allocations. ...allocations are "written" as lockable or not. This makes a difference on how they are handled. It's probably going to be impossible to know how the allocations are written for Skyrim.

Again, the only test I can think of to check out the theory of the game reaching it's RAM limit is to have several people push it to that limit and see if they all have crashes occur.

#70 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 27 February 2013 - 03:32 PM

The way you're highlighted text reads (and this just might be me) is that it depends on how the DX9 application (game) is written to handle the allocations. ...allocations are [i']"written"[/i] as lockable or not. This makes a difference on how they are handled. It's probably going to be impossible to know how the allocations are written for Skyrim.

Again, the only test I can think of to check out the theory of the game reaching it's RAM limit is to have several people push it to that limit and see if they all have crashes occur.

I'm assuming for a moment that the RAM limit crash does exist.  What I'm curious about is whether it might be the Skyrim engine deciding to cache the textures in RAM (in which case there is the slight possibility of this being configurable) or whether this is certainly an inescapable behaviour of DX9.  There might be a techie around here who can testify.


  • 0

#71 viking

viking

    Prisoner

  • Members
  • 0 posts

Posted 27 February 2013 - 03:41 PM

techangel85: Thanks for the clarification. I'll probably just let a mod change the title from now on. I agree with your testing idea. We need a couple of people, besides Neo and myself, that have 6GB+ RAM, to help test the limit. The cheese wedge idea is interesting, but I'm not sure how to implement that. If we would rather go the mod way, we need to determine several Ram heavy/Vram light mods, such as Interesting NPCs, that we can use. Any other recommendations on how we can run this?
  • 0

#72 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,623 posts

Posted 27 February 2013 - 03:44 PM

If a test configuration can be put together, I'd be willing to test this. I have 8GB of RAM with 1GB of VRAM.

#73 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 03:51 PM

 

The way you're highlighted text reads (and this just might be me) is that it depends on how the DX9 application (game) is written to handle the allocations. ...allocations are [i']"written"[/i] as lockable or not. This makes a difference on how they are handled. It's probably going to be impossible to know how the allocations are written for Skyrim.

Again, the only test I can think of to check out the theory of the game reaching it's RAM limit is to have several people push it to that limit and see if they all have crashes occur.

I'm assuming for a moment that the RAM limit crash does exist.  What I'm curious about is whether it might be the Skyrim engine deciding to cache the textures in RAM (in which case there is the slight possibility of this being configurable) or whether this is certainly an inescapable behaviour of DX9.  There might be a techie around here who can testify.

 

Yes this is how I read it as well. There used to be a full copy of the allocated VRAM in physical memory. With DirectX10 this is handled by WDDM and thus no longer takes up space in the virtual memory allocation.  If an application creates its own in-memory copy of its video resource OR uses Direct X9, they still produce a copy. Memory is lockable or not, in other words, it cannot or can be swapped out and make space for other allocations. The VRAM that is normally tagged as able to be swapped out (unlocked) (hence real VRAM cap is higher than when all VRAM is allocated), is not copied into virtual memory space. Now we need to figure out what is written as locked. Perhaps Jason2112 can help out?

EDIT: I should say written rather than copied into virtual address space. I do wonder how that works, because basically it means that the process can use considerably more VRAM, as it is no longer considered part of the address space....?


  • 0

#74 Salvador

Salvador

    Guard

  • Mod Authors
  • PipPip
  • 128 posts

Posted 27 February 2013 - 03:59 PM

If a test configuration can be put together I'd be willing to test this. I have 8GB of RAM with 1GB of VRAM.

Same here. Gonna test the cheese wedges right now. See how much mem they take up in vanilla skyrim.


  • 0

#75 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,623 posts

Posted 27 February 2013 - 04:02 PM

Seems like the 4GB limit has also be confirmed by others on Nexus: https://forums.nexus...sage-increases/ EDIT: After Googling and reading some more, I'm convinced that this CTD issue is caused by Skyrim reaching it's RAM limit. We only need to confirm by a few more testing. Viking and Neovalen have both already confirmed this issue at around 3GB of usage already. The fix for this would only be to decrease your mod load.


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users