Jump to content


Photo

Ramifications of BSA Extraction in Mod Organizer


  • Please log in to reply
270 replies to this topic

#31 Kelmych

Kelmych

    Dragon King

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 3,668 posts

Posted 22 May 2014 - 08:41 PM

It might be relevant to some of the mods in the

" A Real Explorer's Guide to Skyrim" pack. Some of these mods have BSAs with resources that overwrite vanilla assets and some STEP mods that have low priority values, and I doubt we always want the versions of these assets in the BSAs.



#32 DoubleYou

DoubleYou

    Wiki Stepper

  • STEP Staff
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,463 posts

Posted 22 May 2014 - 08:54 PM

 

"BSAs may also be unpacked in this tab by right-clicking the BSA and selecting Extract... This will extract the BSA's contents to any folder you choose. An accompanying ESP file to extract the BSA is not necessary in Mod Organizer. The only requirement is for the BSA to be checked here. "

 

This is not very clear, since the ESP does not extract the BSA ... or is the BSA effectively extracted when the BSA is checked? .... I hope you see what I mean: there are ways we can be stating things that are more explicit and clear. The above leaves questions and even invokes new ones and is not isolated among the doc.

Sorry, that was my fault. Must have been a little confused that day. It should read "An accompanying ESP file to LOAD the BSA"



#33 DoubleYou

DoubleYou

    Wiki Stepper

  • STEP Staff
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,463 posts

Posted 22 May 2014 - 09:25 PM

 

@DoubleYou (and Tannin or whomever)

I am afraid that I am still not clear on how MO handles BSAs. If I take the explanation about BSA priorities in the MO guide, then

  • It would seem that MO can prioritize BSA content (as a solid block of assets) according to the install prioritization rather than the plugin prioritization, thus, decoupling a BSA from its plugin. [color=#ff8c00;]Yes, entirely, if the BSA is checked in the Archives tab. I would not say that it treats it as a solid block of assets, however, as a Mod Organizer proactively scans BSA/loose file conflicts and doesn't load loose files that would provide asset management contrary to mod priority order as perceived via the data tab.[/color]
  • When a BSA in the Archives Tab is checked, it allegedly loads according to install prioritization, but if it is unchecked, it is loaded according to its plugin prioritization [color=#ff8c00;]Yes [/color]... this does not jive [color=#ff8c00;]??? [/color]at all with the caption on the bottom of the Archives Tab window ("Marked archives ( :!:) are still loaded on Skyrim but the regular file override mechanism will apply: Loose files override BSAs, no matter the mod/plugin priority") [color=#ff8c00;]I think you are confusing MARKED-with-warning-icon BSAs with CHECKED BSAs[/color]; I only get the :!: symbol if the BSA is unchecked and has a plugin that is checked in the Plugins Tab. [color=#ff8c00;]Correct. [/color]Otherwise, the BSA is either checked or unchecked [color=#ff8c00;]with a deactivated plugin[/color]. Furthermore, right clicking the BSA provides an option to extract. [color=#ff8c00;]Very useful feature.[/color] Finally, dummy plugins used only to load a BSA are potentially redundant [color=#ff8c00;]entirely unnecessary[/color], but otherwise not [color=#ff8c00;]if the BSA is unchecked in the Archives tab[/color]. This is all very confusing, so we need a very clear explanation of the four scenarios and if any of the four scenarios following do or do not apply if extract is/is not used AND/OR if the plugin is a dummy or not:
    • Plugin checked, BSA checked - ? [color=#ff8c00;]Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.[/color]
    • Plugin checked, BSA unchecked - :!:  ... ? [color=#ff8c00;]Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." Checked BSAs will NOT overwrite if I'm not mistaken, which is why unchecking BSAs can be dangerous.[/color]
    • Plugin unchecked, BSA checked - ? [color=rgb(255,140,0);]Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.[/color]
    • Plugin unchecked, BSA unchecked - ? [color=#ff8c00;]As if the plugin and BSA never existed.[/color]
  • Is checking/unchecking in the Archives Tab an alternative to BSA registration or is this effectively accomplished by registering the BSA in Skyrim.ini? [color=#ff8c00;]Yes and no. This goes back to the the Nitpick fix I think, and is a hack that acts like injection into the ini but is not actually doing so, if my memory is serving me correctly. BSA registration is definitely not an alternative to checking it in the Archives tab.[/color]
  • Does MO scan BSA archives and include the asset conflict information in Mod-Right-Click > Information > Conflicts? [color=#ff8c00;]Yes, except archives located in the actual data directory.[/color]

 



#34 fireundubh

fireundubh

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 350 posts

Posted 22 May 2014 - 10:22 PM

The only time we extract a BSA is if we need a lower priority BSA. Then we could hide the extracted file(s) as needed. Also, texture optimization, but we can get around that by uploading pre-opted BSAs.

I'd also wager that SSD read speed is faster than BSA decompression/read speed.

Edited by fireundubh, 22 May 2014 - 10:23 PM.

  • 0

#35 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,375 posts

Posted 22 May 2014 - 10:40 PM

So in other words DoubleYou the only mods we need to extract are texture mods that are higher in priority that we want to hide files in so that lower priority files can take priority?

 

 

Currently I believe there is no STEP mod that requires such treatment, but that could be a potential case, but it wouldn't necessarily need to be a texture. It could be a mesh or other asset as well.

Keep in mind that we so have assets in STEP that have instructions to hide certain files and this is a feature that might become move important as STEP grows. Whether these are BSA assets or not, I couldn't tell you without checking every single one because I am one of those users that extract all BSAs. This is mainly a habit from Wrye Bash days where if you didn't extract them, you wouldn't be able to see the conflicts which is needed for testing/troubleshooting purposes; not just for STEP (which it is greatly important for us) but also for the community as a whole. If MO allows for proper conflicts to be registered (in the conflicts tab) for BSA assets then there is little reason to extract BSAs since MO essentially treats them as being loose files and the installation loader (normally) holds the keys to which assets are loaded in which order.

 

The exception to this and when BSAs should be extracted (from my point of view) are:

 

When the BSA contains assets that need to be hidden. This is necessary because the file tree tab (where you can hide these assets) will only show the BSA and its ESP. Thus, extraction will be needed to hide files. If Tannin were to ever be able to add a way to hide assets within BSAs without extracting them, then is one exception would go away and there would be no reasons (that I can think of) to extract BSAs while using MO.



#36 EssArrBee

EssArrBee

    Incompatibilism Manager

  • STEP Staff
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7,628 posts

Posted 22 May 2014 - 10:49 PM

I'd also wager that SSD read speed is faster than BSA decompression/read speed.

Not sure what you talking about. SSD vs HD, obviously SSD is faster, but once you get to DRAM all the up the ladder to registers that wouldn't even matter. If you are talking about retrieving data from disk memory then you add like 5 clock cycles for decompression if it is considered native. I'm assuming that the Creation Engine has native support for decomp on Bethesda Software Archives, so the difference is negligible. It used to be something we worried about with Oblivion because it was single core dependent, but with multi-core support there is no longer a need to worry about the difference. 



#37 phazer11

phazer11

    Chatroom Supervisor

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 3,250 posts

Posted 22 May 2014 - 11:12 PM

The only time we extract a BSA is if we need a lower priority BSA. Then we could hide the extracted file(s) as needed. Also, texture optimization, but we can get around that by uploading pre-opted BSAs.

Yes, for the STEP guide only, for packs like REGS or Skyrim Revisited where you have to optimize textures and such to play them without overload I'd think it's a bit much to expect the guide authors to do.



#38 Aiyen

Aiyen

    Dragon King

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 3,536 posts

Posted 23 May 2014 - 03:38 AM

Afaik then the only way one can reasonably come up with the "performance" argument is if a user is on a PC where the CPU is rather old but is still able to run the game. More so if you have an SSD and or low amounts of RAM. However I am fairly certain that this does not apply to the vast majority of the player base. Even when one include the laptops. 

Even then most of the difference is in load times, and is measured in a few seconds tops... unless something is really bogging down the system. 

 

The only reason to do extraction is for convenience if you are going to alter assets. As for repackaging back when you are done for performance reasons then the above still apply.  

 

In general if one have issues with loading times and massive amounts of stuttering then it is much more likely that one is running way too many textures and high detail meshes. 

 

In fact I think the only reason some people still use this argument is because it has been around since oblivion where the PC´s where not as powerful, and where the differences where noticeable. 

 

All that said then nice to see that this discussion is producing some nice information! 



#39 Kuldebar

Kuldebar

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 754 posts

Posted 23 May 2014 - 04:28 AM

I don't like archives in my mod folders because I want them readily searchable via a simple file name search and laid out in such a way that if I want to hide one file, or replace it even; I can do so.

 

I have had several problems with USKP "fixes" that broke something in my game. If I hadn't known about BSA extraction, I'd never resolved the issue by myself. Curiously, Arthmoor wanted to blame MO's BSA handling for the problem which it was certainly not the case! And here was the fix:

 

 

QuoteQuote

So, using MO, (proudly using MO!) I used the "hide" function on the mesh and texture that caused the problem:

"C:Mod OrganizermodsUnofficial Skyrim Patchmeshesactorscharacterfacegendatafacegeomskyrim.esm0001367c.nif"

"C:Mod OrganizermodsUnofficial Skyrim Patchtexturesactorscharacterfacegendatafacetintskyrim.esm0001367c.dds"

 

[color=rgb(255,140,0);]Just to be clear: these files weren't overwriting anything, they just were causing a problem from their very existence in my mod order. [/color]

 

This issue has persisted through the last two USKP releases, each time I have had to hide those two files or an NPC in Half-Moon Mill gets a bad face lift.

 

Without the USKP Fix:

Posted Image

 

 

With the USKP "Fix"

Posted Image


Edited by Kuldebar, 23 May 2014 - 04:41 AM.

  • 1

#40 Kuldebar

Kuldebar

    Jarl

  • Members
  • PipPipPipPipPipPipPipPipPip
  • 754 posts

Posted 23 May 2014 - 04:36 AM

 

 

In fact I think the only reason some people still use this argument is because it has been around since oblivion where the PC´s where not as powerful, and where the differences where noticeable. 

 

 

The "argument" still has merit, compression adds another layer of work for the CPU to deal with, even if it is a relatively small work load by modern standards. The same principle is involved with the much discussed [color=rgb(64,224,208);]EnableCompression[/color] setting in the ENB config file; the consensus is that compression can cause stuttering/slowdown and performance penalties for some computer systems.


  • 0

#41 Tannin

Tannin

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 342 posts

Posted 23 May 2014 - 05:54 AM

The "expected/intuitive way" depends on the perspective of the user. That may be why I have been getting confused and seeing mixed messages.

We are sooo going in circles here... There are no mixed messages in MO. There are mixed messages between posts on the internet that don't discuss MO at all and what MO does. You are confused because you desperately try to apply knowledge that is completely unrelated no matter how often I tell you "MO doesn't work like Lojack said when he wasn't talking about MO but MO works like MO tells you how it works!" MO has a conflicts view. There it tells you which files conflict in which mod. And it gives you a warning if something you configured makes the conflicts view display wrong information. One line. And this is the truth, this is the reality and NOTHING you have ever read outside of MO is relevant to how asset ordering works in MO. There is one single line of information you have to read and comprehend to know everything you need to know. In standard Skyrim asset priorization you have to know which types of asset-containers are applied in what order, you have to know your installation order and you have to know your plugin order, all of which lojack documented in a lengthy post and from that information you can deduce which assets get used. In MO you don't have to know this stuff, you just look at the conflicts view. And therefore, since there is no need to "know" technical details to understand what happens we don't need a lengthy documentation. All it could ever do is confuse the users, make them think there is something complicated to necessitate hundreds of lines of documentation, make them believe there is difficult stuff they need to know when there really isn't.

This is effectively like not installing the mod; however, the BSA could still be registered in Skyrim.ini, but as I understand it, MO does not effectively register BSAs via INI tweaks, right ... ?

When did ini tweaks become a part of this discussion? Forget ini tweaks, they have nothing to do with anything. MO virtualizes the ini file as well as FS. When you allow MO to manage a bsa that bsa will be loaded through the (virtual) ini file but this is a technical detail, you don't care about it! And please stop talking about "registered BSAs"! That is a term lojack used and is explained as "BSAs that have priority lower than plugin-loaded BSAs and loose files". This is not how BSAs work with MO. They are loaded in the same way but that is a technical detail, you don't care about it!

If I understand correctly from your explanation above, the standard load mechanism and prioritization are in effect when the plugin is checked but the BSA is not: The BSA is loaded with the plugin, and the BSA assets override other plugin-loaded BSAs loaded previously (as well as registered BSAs), but loose assets still override the assets within this BSA, regardless of the install prioritization of the loose asets (this is the standard behavior outside of MO). Is that correct (if it is, I will propose a better :!: info text to avert confusion).

Yes it is correct, no it's NOT a better info text because it explains how MO isn't supposed to work! Why would I document inside MO how MO doesn't work?

This is another big point of confusion for me. Are you saying that MO adds an INI tweak to Skyrim.ini to register the BSA? I assume NO. I assume that MO loads this BSA according to the install prioritization as described above and that BSA registration is not even involved ... ?

What I'm saying is: that is a technical detail, you don't care about it!

OK, so NONE of the Skyrim BSAs' assets (including the official DLC) are exposed within the conflict resolution of the mod Information dialog box.

And why would they? IF the vanilla BSAs were exposed then every single asset that replaces something in the game would be in conflict! And, since you can't modify the "installation order" of the original data directory you couldn't do anything about it.

But removing elements of control punnishes those that understand the functionality (or those that want to, are trying to, and are wise enough not to bother mod authors for their own lack of understanding).

While true, I do this in my spare time, I don't get paid for my work on MO. Satisfying the users, while important to me it is NOT my primary concern. And while user groups usually find reasons why they should be more important to the dev than others, they aren't right. If a feature is mildly useful for user A but a major annoyance for user B I'll remove/replace it, no matter how much user A thinks he is more important.

Certainly, your testers (the MO and STEP staff among them) need to remain wary of what happens if things are not working correctly and to recognize any signs of problems. And we need to boldly emphasize certain talking points that address anything controvercial or not well understood.

"not working correctly" means "working different from how it was designed" not "working different from how someone expected it to work". Testers, like regular users primarily need to understand how MO is supposed to behave so they can recognize a deviation but they don't need to know how it is implemented.
  • -1

#42 Tannin

Tannin

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 342 posts

Posted 23 May 2014 - 06:03 AM

The "argument" still has merit, compression adds another layer of work for the CPU to deal with, even if it is a relatively small work load by modern standards. The same principle is involved with the much discussed [color=#40e0d0;]EnableCompression[/color] setting in the ENB config file; the consensus is that compression can cause stuttering/slowdown and performance penalties for some computer systems.

compression adds work to the cpu but lessens the work on the disc. But CPU speed has increased much much faster in the last couple of years than disc speed. Therefore compression on an even remotely modern system, basically anything able to run skyrim, compressed BSAs are way more likely to reduce stuttering than increase it.

The enb topic is completely unrelated. BSA data is compressed on disk and uncompressed very rarely (i.e. on startup, when switching zones).
ENB data, if I understood the point of that setting correctly is about data compressed in memory and uncompressed every frame.
  • 0

#43 fireundubh

fireundubh

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 350 posts

Posted 23 May 2014 - 06:32 AM

While somewhat OT, the only real way to determine whether loose files load faster than a compressed BSA on a SSD is to run that test. I have an Intel Core i5-3570k @ 3.4Ghz, 16GB DDR3-1600, and a Samsung 840 EVO SSD. The performance gain, if any, from loose files is probably negligible, but the potential placebo effect is to-die-for!
  • 0

#44 GSDFan

GSDFan

    Jarl

  • Moderators
  • PipPipPipPipPipPipPipPipPip
  • 788 posts

Posted 23 May 2014 - 06:50 AM

[font="calibri, sans-serif;"]@z929669[/font]

 

 

One point though is that all of the Skyrim - * BSAs are greyed out (untoggleable) and checked (these are registered in the Skyrim.ini), but official DLC BSAs are not grayed out and behave like any other add-on BSA (these have plugin loaders and are NOT registered in Skyrim.ini, nor should they be). A brief word about the greyed out registered Skyrim - * BSAs would be helpful.

This is controlled in the settings under Wordarounds, Force-enable game files. Removing the check will set them be able to be toggled in the data tab.



#45 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,375 posts

Posted 23 May 2014 - 08:27 AM

[snip] but this is a technical detail, you don't care about it! [snip]

This statement... It's a bit unrealistic to think that some users (however small a group they are) aren't going to "care about" how MO works on some of the more "technical details". These are the "power users" and technical minded individuals which make up the majority of the STEP Admin/Staff/Moderators as well as the "old school" modders. S4N and Z are two of the most logical and technical minded people I've ever known. This small group of people are going to want the documentation and are going to want to know the technical details because it's in their very natural to do so and you can't change human nature.

 

I understand MO is designed in a way to make it super simple to mod for the majority of users without having to know a lot of technical aspects of modding. This is the reason it has been a more popular platform for new modders while many of the old modders tend to want to stick with WB (even though its develop has gone six feet under). You've done a brilliant job making MO simple for users with the black magic you coded into it! Well done on that!

 

However, the majority of the "old school" modders are going to have all the knowledge and "old ways" instilled into them from years and years of modding games. Asking this group to "not care about the technical details" is akin to "teaching an old dog a new trick"...it's nearly impossible, if not completely impossible. (You have no idea how long it took the whole staff to convince Z to swtich from WB to MO... :lol: ). Because of these old-way-modders and the logical and technical minded people, it will make it much harder for them to simply use it "because it works" without knowing the details behind why and how it's working compared to new users who are just going to use it...because it works.

 

I hope that sheds some light on the "whys" and "hows" that you are getting from Z and other modders that fit into that group (btw, the Unofficial Patch teams fit into that group as well which is likely why they have some negativity towards MO...most of the "power house authors" are "old school modders").




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users