Vurt's stress test and STEP:Extended

[Spock]

For everyone not knowing what Vurt's stress test is:

Open the console and enter the commands tcl (toggle clip) tgm (god mode, optional) and player.setav speedmult 1000

Then fly around the world very fast loading lots of stuff stressing the engine.

 

 

I would like to know what your results are with STEP:Extended. My game usually crashes at 5-8 minutes. This might be due to my GPU clocking down when the game is loading assets. What do you think about the stress testing method to asses your game stability?

 

Thanks in advance for any input on that matter,

 

Spock

[Freyrgjurd]

I think is a little overkill, since you'll not have, at anytime (IMO), that amount of cells and resources loaded simultaneously.

 

Mainly, I go to a main city, interact a bit and walk a lot. FTravel to another main city and do the same.

 

Windhelm -> Solitude -> Whiterun -> Markarth.

 

It will stall or crash soon enough if the game is unstable, at least for me.

[TechAngel85]

I can go about the same about of time but it's a complete bogus testing method. You'll never stress the game to that point at any time during normal gameplay. Does crashing doing this test mean your game is unstable? No. It simply means you're pushing the engine to do something that it was NEVER designed to do. The engine is flaky enough without users doing dumb stress tests on it. I don't really care if Vurt can do it without crashing. From my understanding from interactions with him on these forums, his game isn't as heavily modded as most player's games are anyway. I put no merit in that test of his simply because I can crash after 5 minutes of doing it but yet can play all day long, for hours on end without a single issue or crash and can do this time and time again.

[hishutup]

That exactly what I was going to say. In theory it'll work but in reality its doesnt...

The best way to test is to literally play the game for an endurance run. 

In one test I purposely played the game for 12 hrs and not a single crash.

 

What was my LO, I dont remember, it was in the days of when I used NMM

[O0mix0O]

I usually go to solitude and spawn 10 dragons at a time until I reach about 50 of them. Watch my fps drop, then just keep adding more until I'm down to single digits, then leave it to run for 10 minutes.

[hishutup]

that still doesnt do it... there could be an error for a spawn or a lvled list and you would never know

[camaro_69_327]

I would like to know what your results are with STEP:Extended. My game usually crashes at 5-8 minutes. This might be due to my GPU clocking down when the game is loading assets. What do you think about the stress testing method to asses your game stability?

 

 

 

I DO use the Stress test as a way to know for a fact i can load just about the entire game , and all areas without crashing. My last setup , STEP + Extended + 150 other mods (253 total) 2K HD...bla bla.. I got bored at 45 mins , covered teh entire game map a few times , never crashed. Went on with that game setup for 170 hours , level 38 , played many many 8 hour sessions...As a Stability test ..I trust it.

SPM Graph (45 min stress test.)

 

Fly low , Fly fast...lol

[Searcher7]

I also use Vurt's stress test. I somewhat disagree that it is invalid because a "normal" game will never undergo that level of "stress". Stress testing is a common method used to identify "weak links", rare failure modes, accumulating errors, and other issues not identified at design time. If the process under test were actually purely deterministic, the tools used to construct the PUT were flawless, and the original design perfect, I might agree. However, none of that is true. Operating systems are flawed, compilers are flawed, programs are flawed, hardware exhibits error rates, and users take unanticipated actions.

 

Because of the inherent, and unknown, flaws present in the system, a heavily modded Skyrim game is a stochastic process, IMHO. The best you can hope for is a low (statistical) probability of failure in an environment arrived at through empirical testing. Obviously, Bethesda missed that boat in the QA, alpha, and beta testing stages for Skyrim. Yourselves (STEP), and the modding community in general, have "tweaked" the environment of Skyrim over the years to a fairly low probability of failure.

 

Vurt's Stress Test is valuable insofar as it can exercise a great deal of game circumstances in a short amount of testing time. Repeatable errors are quickly identified and solved in a traditional manner. Random errors during the test, or in game play, are problematic for many reasons, not the least of which is the lack of official game specs. Trial-and-error solutions become the norm and the time advantage of Vurt's Stress Test helps in this scenario.

 

In my case, with a lot of help from Garfink's older testing threads, I stumbled upon a repeatable result in the Stress Test which has a high correlation to my particular game stability; texture sizes. With a VERY reliable game setup, including most proven stability enhancements, my games will CTD randomly in, roughly, 10-15 minutes of stress testing. IF I optimize textures (DDSOpt) to 1k diffuse and 512 normals, the CTD's go away. Like camaro, I can zoom around Skyrim until I'm sick of it without crashing. An unoptimized texture game will randomly CTD about every 20-30 hours of gameplay (1/month). An identical, but optimized game simply does not CTD (0/7 months). I thought this might be a particular hardware/software/karma issue until I repeated the results on a new game machine. I have no idea why this is true on my machines/games; it simply IS.

 

Don't discount testing methods off-the-cuff. Vurt decided upon a particular test for SOME reason(s). It SEEMS to have helped me find an issue with my particular setup. Of course, YMMV.

[IdiotDave]

I think this is relevant to this thread from the MMO Forum:

@hishutup

Really? I was about to start stress testing using both Vurt's and another where you run around in god mode at a fast speed (but not as fast as flying around). big loops, trying to load most areas of the game until it either crashes or my wife tells me that I have to choose, her or skyrim....

 

Anyway, can you (1) give us the low down on why it doesn't test anyting? and (2) recommend a good test(s) for stability?

Thanks in advance.

 

It the whole vurts thing was just talked about

Vurt's stress test and STEP:Extended

---

 

All it does is stress the engine and isn't reliable or consistent.

That reliable or consistent part is quite critical.

and I am slightly biased

 

If you are testing several NEW mods and you use the vurts test and follow a specific path and receive a crash after 20min.

the second time 20min

the third 8min

the fourth 1hr

...

all with the same LO

this shows that it isn't consistent at all

---

Now my experience with it (this was back when I didnt really understand what I was doing and without memory patch 3.0)

I tested several times

First was 30ish min

Second was 10ish min

third 30ish min

the fourth 2-3hr

 

So I figured I was good... 

The game crashed after 10 min of playing normally

 

 

If you want to know what did it...

It was errors in the Lore friendly version Monster Mod

It happened around the same location but wasn't precise at all

I just discovered this a few months ago :/

 

 

 

---

What would I say to test.

 

oh... this is difficult and I don't have a perfect answer...

So, this is what I have found and many may disagree...

Maybe look for a testing method topic and get some people debating on it or it may have been discussed previously.

I'll look later and make a new post

 

Literally, walk around and play the game mindlessly.

I generally watch youtube vids but most users games should be fine as long as memory patch 3.0 is working, mods are patched in house, no errors, not maxing VRAM or pushing your system too far.

 

I have not used Vurt's or similar methods on a consistent path, nor have I tested for repeat-ability of the test. I have used it on set ups that crashed often during normal gameplay (once every 2 to 3 hours) and on very stable games (8 to 12 hour sessions, no crashes). The length of time I run in a Vurt's style stability test seemed to correlate to the stability of the game. So I didn't think about it much beyond that.

 

However I think Hishutup brings up a good point. Is it repeatable? Look, I'm an engineer leading a diverse team who works on large complex systems. We do a lot of stress testing, and Highly Accelerated Life Testing (HALT). When we design and validate these tests we look at many aspects.  The 2 most relevant (IMO) to this discussion are (1) do we get repeatable results? and (2) Does it reproduce issues (or wear) seen by our customers using the product?

 

 

I also use Vurt's stress test. I somewhat disagree that it is invalid because a "normal" game will never undergo that level of "stress". Stress testing is a common method used to identify "weak links", rare failure modes, accumulating errors, and other issues not identified at design time. 

I completely agree. This is the value of stress testing. The point isn't to put the product under a 'realistic load' but under a load that will accelerate real (sometimes rare) failures. If it takes us 30 to 40 hours of regular game play to identify a major stability issue in the game, it isn't practical to try to fix it, then play 40 hours only to find out you didn't.

Vurt's Stress Test is valuable insofar as it can exercise a great deal of game circumstances in a short amount of testing time. Repeatable errors are quickly identified and solved in a traditional manner. Random errors during the test, or in game play, are problematic for many reasons, not the least of which is the lack of official game specs. Trial-and-error solutions become the norm and the time advantage of Vurt's Stress Test helps in this scenario.

Searcher 7, what do you mean by repeatable errors? Is this an error you get after a consistent amount of time running Vurt's Stress Test, or an error you get when you consistently load the same cells in a specific order quickly? Or a location the consistently crashes but seemingly at random (doesn't seem to rely on amount of time spent testing, or order of cells loaded, etc. only the general area the CTD happens)?

 

Has Vurt's or any other stress test method helped anyone find or diagnose a problem in their regular gameplay?

Edited November 26, 2014 by IdiotDave

[Aiyen]

We had a long debate about vurts way of doing it a while back... I am in no way an advocate for it since in short it is not a good test for more complex mod lists. It can work for less advanced ones and if you just want to test bandwidth.. which is highly relevant for vurts mod since it does add a lot of extra polygons to the calculations. 

 

 

In general there are no 100% reproduceable and reliable stability tests... since all mod lists and hardware are unique. 

 

Also skyrim can and most likely will always CTD at some point! Due to the complexity of the game it is simply unavoidable. The main thing the user want to achieve is hours worth of playing in one go. 

 

However what the tests like the one vurt does is good for beyond testing if your rig can handle the bandwidth is finding basic errors. After you have done all the tes5edit patching to make sure there are very few to no inconsistencies in your load order then you can fire up this sort of test and see how it does. 

 

My own test is the following. 

 

Movespeed at 500 start in riverwood. 

Run past falkreath to markarth to solitude to dawnstar to winterhold to windhelm to riften and back to riverwood. 

 

The reason for using movespeed 500 and no flying is because if you fly then you tend to move beyond the cells and do not load anything anyways... also the AI does not really like flying it really does mess it up (try to see how dragons behave when you fly.. quite hilarious). All this can and will produce false CTD´s. Another reason for not going faster is that you will tend to overshoot several cells in a row hence the game never actually loads your content at all.... which is again kinda counter productive... also it can provoke CTD´s since the game starts some processes which require certain things to be loaded. 

 

My criteria for a successfull test is that I can get past dawnstar three times in a row. After which I will try to do the test in reverse as well. In general this will allow me to cover a large portion of the game world, and see quite a few random encounters, and events and spawns... all things which should just work if the conflict resolution have been performed perfectly. 

 

If the list is really complex I will also do the same at higher player levels in order to check the high level versions of the content. At higher levels you will have more diversity on most lists and it is again vital to check so you do not end up with a borked game at lvl 40. 

 

Ofc. this is not the only test I personally do.. I got plenty of other hot spots for trouble I go and check out and see how they go as well. In general it takes me about an hour or so to stress and stability test. 

You can ofc. ask what all this is good for if it is not 100% reliable... well most common issues will be rooted out and if you for example get a CTD in the same spot over n over, or in the same general way (IE you get a CTD where you know a spawn will appear) then you can reduce the amount of records you will have to double check in tes5edit. 

 

 

That was a bit of a long and semi complex rant... hope it was somewhat useful. 

[TechAngel85]

My own test is the following. 

 

Movespeed at 500 start in riverwood. 

Run past falkreath to markarth to solitude to dawnstar to winterhold to windhelm to riften and back to riverwood. 

 

The reason for using movespeed 500 and no flying is because if you fly then you tend to move beyond the cells and do not load anything anyways... also the AI does not really like flying it really does mess it up (try to see how dragons behave when you fly.. quite hilarious). All this can and will produce false CTD´s. Another reason for not going faster is that you will tend to overshoot several cells in a row hence the game never actually loads your content at all.... which is again kinda counter productive... also it can provoke CTD´s since the game starts some processes which require certain things to be loaded. 

 

My criteria for a successfull test is that I can get past dawnstar three times in a row. After which I will try to do the test in reverse as well. In general this will allow me to cover a large portion of the game world, and see quite a few random encounters, and events and spawns... all things which should just work if the conflict resolution have been performed perfectly. 

 

If the list is really complex I will also do the same at higher player levels in order to check the high level versions of the content. At higher levels you will have more diversity on most lists and it is again vital to check so you do not end up with a borked game at lvl 40. 

This, IMO, sounds far more reliable than Vurts version. Aiyen addressed most of the issues I have with Vurt's test: you never "fly" in the game, flying causes other issues that Aiyen pointed out, etc. The other major reason is what others have pointed out as well, that it is rarely consistent. CTDs usually happen at random which leads one down a path and complete insanity trying to figure out the source of the crash because it's never in the same place twice (btw, this would most likely be bandwidth issues or you've just overstressed the engine and what it can handle...neither of which is necessarily a "mod" issues). I'm not saying there is absolutely no merit in it. In the case that it does cause a repeatable CTD in one location, then it can help identify an issue; however, I by no means recommend users to include this stress test as part of their troubleshooting procedures. 9/10 times the issues can be found with other, more reliable methods.

[Octopuss]

I used to vurt my setup a lot in past, but stopped doing so because of completely inconsistent results. I assume the Skyrim engine plays a large role here. I could fly around for 10 minutes like a madman once, and then get a CTD within 15 seconds twice in a row. That kind of makes it useless to draw conclusions.

[Searcher7]

 

...

Searcher 7, what do you mean by repeatable errors? Is this an error you get after a consistent amount of time running Vurt's Stress Test, or an error you get when you consistently load the same cells in a specific order quickly? Or a location the consistently crashes but seemingly at random (doesn't seem to rely on amount of time spent testing, or order of cells loaded, etc. only the general area the CTD happens)?

...

 

The latter, primarily. I also "visit" cities and other interior locations when running around Skyrim like the Flash. A particular modded spawn point, for instance, might CTD each time I approach it. I also find issues with missing meshes and textures, even a mod's NPC not being there. Glitches based upon a certain location are more easily found in high-speed testing than normal speed game play. By repeatable, I just mean in the same location/cell each time. Those are the (relatively) easy mod problems to fix.

 

Random problems occur in different locations at different times during the test and I can't replicate them from one test run to another. The (supposed) texture issue I mentioned was a case like that; the game just drops to the desktop in different places and at different times. Very hard to isolate the cause of those type problems. Its why Sheson's patch didn't show up on 12/11/11.

 

As I said, trial-and-error becomes a useful, if not methodical or scientific, method when your understanding of the process under test is limited. We have limited understanding of the Skyrim/Creation Engine code. That makes it difficult to even develop a methodical test suite; I mean, we're still talking about it four years after release. Vurt's test just allows you to speed through an otherwise time-consuming check of game operation. If we can't run a dedicated test suite, might as well see if it will just run. Its not a rocket science based test, just a quick check on overall game operation. In my experience, though, there is a definite correlation (NOT proof) between Vurt's Stress Test reliability and actual game play reliability. Increasing Time-To-CTD (TTC) in Vurt's test has, as far as I can tell, also increased TTC in my games, be it from removing a shaky mod, changing an ini tweak, updating drivers, etc, etc. On my PARTICULAR game machines, there is a strong reliability correlation between Vurt's test results and normal game play. That's all I'm saying.

 

I absolutely HATE glitches in my games. At this point, amongst all the other modding tools and practices, including Vurt's Stress Test has allowed me to mod (my heaviest modded game is STEP Core+REGS+Difficulty Mods) many enjoyable, (almost) error-free games. FWIW.

 

EDIT: Spelling

Edited November 26, 2014 by Searcher7

[mikegray]

Deviating from the OP a little here, but how about this test?

 

https://forums.nexusmods.com/index.php?/topic/1277531-28-step-stress-test/

 

It seems more reasonable to me - though I suppose it's closely related to stuff the SoT mod throws into the mix. I've never had trouble doing that quest in the past - but I've never done it with SoT running ...

 

But this test DOES illustrate that you need various kinds of stress tests to reveal various kinds of weaknesses.

Edited November 26, 2014 by mikegray

[TechAngel85]

You can probably eliminate 20 of those steps by using COC.

[Aiyen]

Well i guess I should post mine in full detail 

 

from the main menu open the console 

 

coc riverwood (no need to use live another day or other mods... this is the fastest way to get ingame and it does not matter if you start properly or using this way... any CTD´s in the char generation can be isolated using showracemenu from the console as well.) 

 

Wait for all mods to load... and activate the mods you want to use during the test etc. You can make a save at this point so you do not have to do it all over again. 

 

tgm  (High speed running will cause your death due to fall dmg!)

 

player.setav speedmult 500 (increase move speed to 500 units, trigger by running once) 

 

Using only the main roads as far as possible run riverwood -> Markarth -> solitude -> dawnstar -> winterhold -> windhelm -> riften - > riverwood 

The specific way you decide to take does not really matter... what matters is that you take the same way in each run. 

 

 

Assuming that goes all well and dandy then coc to somewhere out in the open... I normally just use coc whiterunexterior01 ... spawn 10 mages of varying types, and then some saber cats or mammoths etc. Heavy spell casting is a CTD candidate of choice... but this test should show it (If you do not know the formID´s for mages just type "help mage" and use pgup and down to get some NPC ID´s)

 

 

after that coc solitudeavenues 

Spawn 3 dragons while in the city (Does not work as intended with open cities mod... but using that on large lists is a no go anyways) Give your self a daedric bow or a crossbow and go hunting... you should be able to kill all three dragons without the game CTDing on you.

Dragon fights are heavily scripted and depending on mods used some rather complex AI n stuff. This will make sure you will be able to handle three dragons in a stressfull environment... solitude is full of people and dragons love to just go nuts. It will also test spell on hit effects etc. Good stuff. 

 

 

Other areas and regions of interest: 

Dragonsbrigde... known play for CTD´s even in a pure vanilla game. So do not think all is lost if you get one there. Same applies near the ritual stone on the road between whiterun and windhelm. Jorvaskarr.... this place can cause all sorts of weird bugs and issues. So it is always a good idea to check it out. There are plenty of others but those are the ones that I normally use. 

 

 

Ofc. once you think you got it all down then comes then big one... you should be able to start a new game using the default start and using godmode and whichever weapon or spell etc.. you should be able to get to the first dragon kill without a CTD! or at least very close to it! 

 

The intro is a mean badass thing that can smash stuff... hence it is good for testing. Bugs are allowed to happen... CTD´s are not. Most of the events up to the first dragon kill will showcase just about all aspects of the game and will take about as long as the average playing session would be for most people. If you do it without god mode it will take even longer ofc. The reason for god mode is because we do not want the loading of a game to interfere. Keep in mind that some mods will disable god mode after certain effects apply to you.. like learning a power word. 

 

The whole set of tests... excluding the actual playing of the start of the game should only take about an hour or so when you have done it a few times.... also depends on how cooperative the dragons are in solitude. 

 

 

 phew.. another wall of test! enjoy! 

 

edit: each test can be performed by loading the initial save made, or just using coc from the main menu to test... hoping the game stay alive through that entire amount of tests is in most cases wishfull thinking ofc! :)