DDSopt, MO & the ESMs & BSAs

[xExekut3x]

After completing the DDSopt guide and adding the packaged files to Mod Organizer, what else needs to be done? Specifically, what .esm(s) and .bsa(s) need to be deactivated (unchecked), if any?

 

 

[oqhansoloqo]

*BUMP* *BUMP* *BUMP*

 

My question also. It's a shame to see so much data space go to waste. These are some large files: Skyrim - Textures.bsa, HighResTexturePack01.bsa, HighResTexturePack02.bsa, and HighResTexturePack03.bsa (almost 6 GB). Plus the optimized .7z files: STD.7z, HRDLC1.7z, HRDLC2.7z, and HRDLC3.7z (almost 4 GB, and that doesn't factor in if you install them in Mod Manager to make them loose files, which takes up 8 more GB). So we are talking about 18 GB total - that's a LOT of data space. OK, so the .7z files can easily be removed from the "downloads" folder in Mod Organizer, which recovers about 4 GB... that still leaves the 6 GB for the original BSAs and 8 GB for the loose files installed in Mod Manager (14 GB total). It would be nice to have a total of about 8 GB instead, afterall:

 

If the 4 BSAs were composed of all texture files, and we converted the texture files in them into loose files (from the .7z files), then why can't the BSAs be deleted now?

 

I read something about 102 files in the Skyrim - Textures.bsa file that don't end up in the STD.7z file... are there any files like that in the HighResTexturePack BSA files? If not, can those be deleted now?

 

If the original BSA file(s) cannot be deleted, is there an easy way to reduce the size of it/them?

 

What BSA and ESP (maybe the ESMs for DLCs, or not?) files can be deleted from the Data directory and what can't?

 

I looked all over the DDS/BSAopt guides and cannot find answers to these questions.

[Kelmych]

The DDSopt guide describes how to use DDSopt to optimize vanilla and mod textures. The DDSopt guide does not describe how to use a mod management program (Mod Organizer or Wrye Bash) to manage the vanilla resource archives and BSAs once the optimization is complete. For Mod Organizer you will find this discussed in the forums on Mod Organizer, and there is also general information in the Mod Organizer guide on the Wiki. You should be able to find the specific discussions using the search features on the forums. You might also find some useful information for installing the optimized textures and vanilla plugins with Mod Organizer in the Skyrim Revisited Legendary Guide. Note that the instructions for optimization in that guide are different than those in the DDSopt guide; the instructions in the DDSopt guide provide a more detailed selective optimization. Personally I use the DDSopt guide optimization approach and I use the archived optimized vanilla textures (7-zip format) vs. providing the Skyrim - textures.bsa or the 3 HRDLC BSAs to Mod Organizer, and the 3 plugins associated with the HRDLCs are not activated. There are descriptions in the forums on how to install and clean the Skyrim esm plugins, and how to load any DLC BSAs being used.

 

To answer one of the questions there are some duplicate files in the HRDLC which are deleted as part of vanilla texture optimization in the "Repair the Vanilla HRDLC Textures" subsection  here.

 

I expect (or maybe just hope) that someone will assemble the material from the Mod Organizer forums on how to install the vanilla resources using the optimized textures from the DDSopt guide, and put this into either the DDSopt guide or the Mod Organizer guide.

[oqhansoloqo]

I searched the forums, and information is scattered and difficult to find. I can't find anything that really says exactly what to do regarding deviations in processing the vanilla files and cleaning up the original files in the Data directory. I checked out all of the resources you recommended and can't really find an answer in any of them.

 

DDSopt is a confusing program. The documentation internet link in the program's folder doesn't even work. As an example of some of the confusion: I load HighResTexturePack01.bsa into DDSopt to browse it to see what it plans on trying to process (I guess that's what I'm looking at...?). If I scroll down the list of files in the bottom half of the DDSopt program window I eventually come to blue-colored text files. No documentation at all to explain anything... What are these files? They are still checked off on the left side, but they are blue. I have no idea what this means. If these are files that are not processed by DDSopt, why aren't they greyed out or something more obvious? And I have seen greyed out files in DDSopt before, so I know that is another situation that occurs within the program. I guess they are files that are not processed (I have no idea) by DDSopt? And how does the "Batch_Files_for_DDSopt_guide_v2.61-beta" package determine what to do about choosing which files are duplicates amongst the vanilla BSAs - is that what blue files mean (that the blue-text file is a duplicate of another file? There are so many questions, and extremely confusing answers missing valuable information about the basics of DDSopt. Very very confused.

[Kelmych]

I searched the forums, and information is scattered and difficult to find. I can't find anything that really says exactly what to do regarding deviations in processing the vanilla files and cleaning up the original files in the Data directory. I checked out all of the resources you recommended and can't really find an answer in any of them.

 

DDSopt is a confusing program. The documentation internet link in the program's folder doesn't even work.

When I checked I got a message saying that site had technical problems with the database. The documentation on the OBGE site expects the reader has significant background in graphics modeling, by the way.

As an example of some of the confusion: I load HighResTexturePack01.bsa into DDSopt to browse it to see what it plans on trying to process (I guess that's what I'm looking at...?). If I scroll down the list of files in the bottom half of the DDSopt program window I eventually come to blue-colored text files.

There isn't any basic user documentation for DDSopt that starts with descriptions of Windows graphic models (including texture files), BSAs, etc. It was developed primarily for mod developers and users with some experience in how Bethesda games handle graphics modeling.  It isn't a commercial program developed by Bethesda or some other company. Bethesda provides a Creation Kit for editing and developing mods, but it does not provide many of the tools actually needed to work with mods. DDSopt can be challenging to use. Some STEP users have been working on some basic documentation for DDSopt including videos, but these are not yet ready for use. I've looked for some good short tutorials on Windows graphic models but I haven't found any; understanding DDSopt requires a understanding of some of the basic properties of these models.

 

I've never seen blue marked files in the DDSopt window. When I open HighResTexturePack01.bsa and in the lower window browse to an empty folder and either select use folder or select "Save" to save the result as a BSA there are no files in blue. Make sure you are using an empty folder as the destination when using BSAopt or DDSopt to extract or view files from a BSA. Files that have a checkmark will be included if you hit "Process", and ones that won't be are grayed out. You can edit the checkmarks to eliminate files from being processed.

No documentation at all to explain anything... What are these files? They are still checked off on the left side, but they are blue. I have no idea what this means. If these are files that are not processed by DDSopt, why aren't they greyed out or something more obvious? And I have seen greyed out files in DDSopt before. I guess they are files not processed (I have no idea)?

And how does the "Batch_Files_for_DDSopt_guide_v2.61-beta" do about choosing which files are duplicates amongst the vanilla BSAs - is that what blue files mean?

This archive file includes multiple batch files, two of which are used in cleaning the HRDLC by removing duplicates; the instructions for this are in the DDSopt guide. One of the batch files has a manually created list of duplicate files that are used to remove the duplicates.

There are so many questions, and extremely confusing answers missing valuable information about the basics of DDSopt.

 

 

[oqhansoloqo]

DDSopt's website works today. I'll have a look at their documentation and review what you said and see if I can make sense of the color-coding scheme in the program. I'll let you know what I discover.

[oqhansoloqo]

OK, so here is my problem now. If I am trying to create new slimmed-down files from the original vanilla texture BSAs left over in the Data folder after running the texture vanilla BSAs through DDSopt (the ones as recommended by STEP - not including the mixed-file BSAs recommended for BSAopt processing), I need to know which files are considered redundant and which are not. I think I saw someone elsewhere on the STEP forum talk about doing this: they extracted all original vanilla texture BSAs (plus Dawnguard, Hearthfires, and Dragonborn DLCs) into one folder and let them overwrite each other during the extraction process (thus removing the redundancies between them). Then they took all of the optimized files run through DDSopt and through the batch files and copied them on top of the loose conglomerate of vanilla files, thus replacing all of the unoptimized files with the optimized versions.

 

My problem with this method: now all packs are mixed together into one, and everything is now loose. Do you see any problem with this method? My goals being: to have optimized files where possible, to remove redundant files, to have quick file access, and to maximize storage space. I guess I may have to remove the mention of those BSAs in the skyrim.ini file that is used to load them.

[DoubleYou]

Don't remove the BSAs from the skyrim.ini. Loose files overwrite files inside BSAs every time.

[Kelmych]

OK, so here is my problem now. If I am trying to create new slimmed-down files from the original vanilla texture BSAs left over in the Data folder after running the texture vanilla BSAs through DDSopt (the ones as recommended by STEP - not including the mixed-file BSAs recommended for BSAopt processing), I need to know which files are considered redundant and which are not. I think I saw someone elsewhere on the STEP forum talk about doing this: they extracted all original vanilla texture BSAs (plus Dawnguard, Hearthfires, and Dragonborn DLCs) into one folder and let them overwrite each other during the extraction process (thus removing the redundancies between them). Then they took all of the optimized files run through DDSopt and through the batch files and copied them on top of the loose conglomerate of vanilla files, thus replacing all of the unoptimized files with the optimized versions.

 

My problem with this method: now all packs are mixed together into one, and everything is now loose. Do you see any problem with this method? My goals being: to have optimized files where possible, to remove redundant files, to have quick file access, and to maximize storage space. I guess I may have to remove the mention of those BSAs in the skyrim.ini file that is used to load them.

After completing the vanilla optimization a few people combine everything into a single archive file, but most users prefer to create one archive file for each of the original BSAs. The is some redundancy in that the STD and 3 DLC archive files will have lower resolution versions of some textures that are in the 3 HRDLC archive files, but the mod management programs will only provide one version for Skyrim itself. By the way, if you follow the steps in the DDSopt guide and use the batch file for creating archive files after the optimization is completed, you will get one archive file for each BSA.

[oqhansoloqo]

DoubleYou - was/am not sure if the game will fail after not being able to locate a file listed in the skyrim.ini while trying to load up, which is why I was thinking of removing Skyrim - Textures.bsa from that .ini file if I am going to remove it from the Data directory where that .ini file would have expected to find it. I guess you are implying that I don't have to do it then?

 

Kelmych - I did follow the guide (at least I think I did) and got one .7z file per BSA (STD.7z, HRDLC1.7z, HRDLC2.7z, HRDLC3.7z, Dawnguard.7z, Hearthfires.7z, and Dragonborn.7z). Problem is, they don't include all of the files necessary to run the game - just the texture files that were optimized. I tried just deleting their original vanilla BSAs from the data directory to conserve on storage space but the game will not run due to those necessary files that the optimization process seemed to have not transferred into the optimized archive files. Apparently, the game still needs the original BSAs to be in the data directory after following the DDSopt instructions in the STEP wiki because of files included in them that are not in the optimized archive files. This is why I am trying to modify the DDSopt instructions in the STEP wiki. I don't have the storage space on my SSD to spare for the instructions as given in the STEP DDSopt wiki. That is why I am considering trying to extract everything from the original BSAs first all together, then overwriting that mix of loose files with the loose files from the archived files (.7z files) produced from the optimization process described in the DDSopt guide. I don't see how I can keep the data from each BSA separate - they have to be mixed together in order to ensure that redundancies between them are eliminated. Or maybe there is an alternative that I'm not seeing - that's why I am asking. I don't see any other way to do it. I'm wondering if things will work properly after doing it this way. The only alternative I could see would be to know exactly which files were left out of each of the optimized archive files (not including the redundancies that were removed by the batch files, but rather the files within each BSA that were not processed by DDSopt) and to just grab those from each BSA, create new BSAs or archives (corresponding to each original BSA) with just those files, and then remove the original BSAs.

[DoubleYou]

DoubleYou - was/am not sure if the game will fail after not being able to locate a file listed in the skyrim.ini while trying to load up, which is why I was thinking of removing Skyrim - Textures.bsa from that .ini file if I am going to remove it from the Data directory where that .ini file would have expected to find it. I guess you are implying that I don't have to do it then?

 

Just make sure you have all texture files so you don't have a pretty pink game. Why do you want to get rid of the Skyrim - Textures.bsa? Are you that low on space?

 

Edit: read op

You want the textures to only take up 8GB? Well, the only way I know to do that is to make the optimized textures into a BSA instead of loose files.

[oqhansoloqo]

DoubleYou - My SSD drive is only 120GB and Skyrim is not the only thing on it. I have other older slower drives for archived files, but everything installed goes on the fast SSD. I imagine a lot of people don't have large SSDs because they cost a lot more per data storage capacity than HDDs.

 

I was getting a pink intro screen after trying to delete Skyrim - textures.bsa from the Data folder, and that's how I knew/assumed that it had files in it that were not in the STD.7z archive. I thought I read somewhere in the DDSopt or BSAopt STEP guides that there could be problems with trying to re-BSA the optimized texture files and that it's better to leave them loose.

 

So DoubleYou, do you see any potential issues with doing what I proposed in my previous message or do you think it won't work right?

 

EDIT: I'm guessing I should overwrite in this order (BSAs and .7zs extracted into loose files): Skyrim - Textures.bsa > HighResTextures01.bsa > HighResTextures02.bsa > HighResTextures03.bsa > Dawnguard.bsa > Hearthfires.bsa > Dragonborn.bsa > STD.7z > HRDLC1.7z > HRDLC2.7z > HRDLC3.7z > Dawnguard.7z > Hearthfires.7z > Dragonborn.7z. Does that make sense?

[oqhansoloqo]

Huh? I just checked the DDSopt Guide again. I see in there it talks about moving the original Skyrim - Textures and HRDLC BSAs to a folder outside of the Skyrim directory. I don't remember seeing it mention that before. Did someone just change this recently? Looks like it was updated today actually. Nice. I'll have to reprocess my BSAs and follow the new instructions. This is good - it'll allow saving about 5.5 GB in the Skyrim directory by taking those original 4 BSAs out. :)

[Kelmych]

That certainly applies when using Wrye Bash. With MO, I'm expect (but I'm not sure) that leaving them in them in the Skyrim\Data folder causes Skyrim to load these BSAs and then MO overrides them. I'm not sure what MO suggests; I think it prefers that you leave them so the Data folder itself looks that same as when Steam installed it originally. It does save space, as you point out, by moving them out of the Data folder. I don't know whether this causes any problems with MO.

[oqhansoloqo]

So I just went through most of the STEP DDSopt Guide processing for vanilla Skyrim BSAs (Using DDSopt and DDSopt Optimization tabs in STEP DDSopt Guide), but I ran into a potential problem this time. As far as I know I followed all instructions and my settings matched screenshot settings exactly. And I have the full complement of the 3 DLCs that were optimized as part of this. I chose the H1-H3 DDSopt Constraint Menu Settings. I got to near the end of the "Optimize the Vanilla Textures" procedures - finished step 11. At that point I decided to review the "Folders in the Working Directory and Counts of Files and Folder in each Directory" screenshot to see how many files and folders should be in the Vanilla Optimized directory on my computer. The screenshot shows 45,125 files and 1,192 subdirectories. When I review the properties of my Vanilla Optimized directory I see 40,912 files and 1,177 folders. I didn't bother to note the difference in sizes since that could vary depending on compression preferences/settings that were selected (I chose the "High Quality" settings), but shouldn't the number of files and folders be the same?

 

The files and folders counts for the Vanilla Extracted, Vanilla Ordinary Textures, Vanilla Normal Maps, and Vanilla Uncompressed Textures folders match the screenshot. One thing to note is the DDSopt processing of the Vanilla Uncompressed Textures did not take long at all to complete in contrast to the processing of the other two.

 

EDIT: Actually, my SSD reached its limit and is crashing now and acting funny. I have to shift files around to other storage spaces to finish the process. I think I have to re-run step 11 again after I make more space available. I'll update on the progress. These files are HUGE.

 

EDIT #2: It turns out that it wasn't just the Vanilla Uncompressed Textures that didn't finish processing in DDSopt, but also the Vanilla Normal Maps - I had no clue because when I came back to the computer to check to see that the processing had completed there were no error messages or anything of the sort. This was all due to running out of disk space, but DDSopt does not give warning about this. It's not obvious to the end user. Now that I re-ran those two processes I now have the correct amount of files & folders in the Vanilla Optimized directory. Fortunately it didn't take long to re-run the processes because it seems to have quickly glazed over the files already processed and basically started from where it left off. Geez, I hope that the files that it ended on previously are not corrupted though and somehow incompletely processed... I think there should be a warning about this in the STEP instructions for using DDSopt. This whole process requires VAST disk space to complete, especially for SSDs. DDSopt does not give adequate warning or notice of incomplete processes and I think that there is a reasonable expectation that quite a few people are going to run into disk space issues. It's near 40-50 GB. I think that STEP should have some kind of comment about this to notify whoever is trying to follow the DDSopt guide about the space required for the optimization process and how DDSopt will not give obvious warning that it did not complete properly if it doesn't due to space issues. BTW - I'm going to redo the entire "Optimize the Vanilla Textures" part of the guide again just to be sure no files are corrupted.

[Kelmych]

One thing to note is the DDSopt processing of the Vanilla Uncompressed Textures did not take long at all to complete in contrast to the processing of the other two.

Yes, there aren't a lot of files in this folder and most are small.

 

EDIT: Actually, my SSD reached its limit and is crashing now and acting funny. I have to shift files around to other storage spaces to finish the process. I think I have to re-run step 11 again after I make more space available. I'll update on the progress. These files are HUGE.

I've usually used an HDD since the folders are large, and I live with the slower processing. The sorting portion (the pre-optimization batch file) takes 20-30 minutes for me on an HDD. One possibility is to do the preprocessing on the HDD, then copy the "Vanilla Optimized" folder to the SSD and move the 3 temporary folders from the pre-optimization step individually as needed to the SSD for DDSopt processing. The DDSopt processing takes a lot more time than the sorting in the pre-optimization so almost of the value in using an SSD is for the DDSopt processing and for the archiving of the optimized files.

 

I think there should be a warning about this in the STEP instructions for using DDSopt. This whole process requires VAST disk space to complete, especially for SSDs.

You can add a note in the guide about this, or we will the next time the guide is edited.