Jump to content


Photo

Catalog of Existing Solutions | Requirements for Any Tool


  • Please log in to reply
18 replies to this topic

#16 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 116 posts

Posted 30 September 2016 - 06:16 PM

Just post in the xEdit topic or PM Zilav about the odd changes. He's on here from time to time and will see it. If they are bugs, then having him correct those bugs will benefit all users. Else, he'll be able to tell you why the changes are made.

That's a good suggestion, thanks.  I've posted over there.


  • 0

#17 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 116 posts

Posted 17 October 2016 - 06:47 AM

All right, this took a lot out of me (and I took a bit of a "break" partway through because work leaked into my free time), but I've finally got what appears to be a pretty good automated plugin cleaning solution.  On my box, it cleans Update.esm, Dawnguard.esm, HearthFires.esm, and Dragonborn.esm all in about 7 seconds total.  It can be this much faster than xEdit mainly because we have to tell it exactly which records are ITMs and which are UDRs instead of having xEdit do its filter thing to figure this out on its own, but also because we can combine a lot of work that's common to all plugins into one.  Both xEdit and this tool need to catalog the records present in Skyrim.esm and Update.esm so it can properly do the UDR step (for those unfamiliar, UDR means "take references that were deleted in the plugin I'm cleaning, restore their contents and location from the previous version, and make a few well-known changes to them", so unlike for deleting ITMs which can be done independently as long as you have a list of what to delete, we need to have the previous version of everything).  However, since all 4 plugins need to catalog Skyrim.esm records, and the latter 3 need to catalog Update.esm on top of it, we can get savings by reusing the same catalog across all the plugins we need to clean.  It's also kinda cool that as soon as we "clean" Update.esm, we can add it straight to the catalog for cleaning the other plugins.

 

Now that I've got what I wanted out of that incredibly long detour, here's what I see is on the roadmap:

  1. Wrap the new automated plugin cleaning process so that it can be invoked via the XML file.
  2. Figure out the ITMs / UDRs for third-party plugins that also need cleaning and get their data into the XML file as well.
  3. Finish up the last one or two groups from STEP Core.
  4. Investigate automated SkyProc stuff.
    1. This is PROBABLY something I can leverage my new tool to do.
    2. Alternatively, maybe the output files can be distributed, since everything winds up being "static" anyway.  I have to figure out what's appropriate from a non-technical standpoint.
  5. "Automate" Bashed Patch creation.
    1. Almost certainly, this new tool can do this.  I'm probably just going to do a cop-out and define it in the XML file.
  6. Figure out DynDOLOD.
    1. I'm not sure if I'm ready to spend the couple of weeks that this would take to automate "nicely".  Maybe this is where I just give up and write something for AutoHotkey.
  7. Testing, testing, testing.
    1. Especially for the automated plugin cleaning stuff, I had to give up on making the output exactly byte-for-byte identical with xEdit's output.  xEdit has about a decade of history behind it, and I'm just the latest in a very long line of attempts to automate STEP, who's never even written a regular mod for Skyrim.  I want to make sure I prove to myself, at least, that the differences are fine.
  8. Package and distribute.
  9. Keep up-to-date with newer versions of mods.
  10. Implement "Pack" support for things that build on top of STEP Core, starting with a "Pack" version of STEP Extended.

So that's what the future looks like.  Right now, it's actually somewhat usable: you can just run the application with the right command-line args and if you have all the files downloaded and certain versions of the official DLC plugins pre-cooked, it'll set up Mod Organizer with a profile exactly as you'd expect by looking at the XML file, and it'll also create a batch file that you can double-click to start Skyrim without having to think about it.  It's just that there's more that needs to be done behind-the-scenes to make it correct and complete.


  • 0

#18 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,051 posts

Posted 17 October 2016 - 09:52 AM

Sounds like you're making good progress. One thing to note is that I wouldn't bother trying to automate DynDOLOD. Sheson has already made it so simple that users can run it themselves without issues. We're already providing the pre-made files they need (billboards) in the STEP Compilation. It's also best for them to run it themselves because this allows any object-oriented mods the user installs on top of STEP to be automatically gathered up and processed. We wanted to keep DynDOLOD a "custom" process that isn't solely tied to STEP so users could expand and customize to their heart's desire without having to do anything special simply because they have STEP installed.

 

The only way I can see making the DynDOLOD process easier to by creating a batch file or something that will set up the folder structure and possibly the executables for the user. This would eliminate about half the steps present on that mod page.



#19 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 116 posts

Posted 23 October 2016 - 11:22 AM

Sounds like you're making good progress. One thing to note is that I wouldn't bother trying to automate DynDOLOD. Sheson has already made it so simple that users can run it themselves without issues. We're already providing the pre-made files they need (billboards) in the STEP Compilation. It's also best for them to run it themselves because this allows any object-oriented mods the user installs on top of STEP to be automatically gathered up and processed. We wanted to keep DynDOLOD a "custom" process that isn't solely tied to STEP so users could expand and customize to their heart's desire without having to do anything special simply because they have STEP installed.

 

The only way I can see making the DynDOLOD process easier to by creating a batch file or something that will set up the folder structure and possibly the executables for the user. This would eliminate about half the steps present on that mod page.

Fair.  Plus, I'm sure that there are plenty of packs that would require re-running DynDOLOD anyway, so it's probably best to hold off on that anyway.

 

Progress on the above:

  1. Got the process doing all the plugin cleaning, specified in the XML file.
  2. Third-party plugins are cleaned.
  3. All of STEP Core, minus DynDOLOD, is covered.
    1. I do want to add one more "option" group so someone can grab either the "Normal Res" or the "High Res" version of the STEP Compilation, since it looks like the two have identical directory structures.
  4. I can conveniently defer SkyProc for now, because it looks like STEP Core doesn't include anything that depends on it.
  5. Bashed patch is "automated", in the sense that I just plopped the entire binary contents into the XML file, base64-encoded.
  6. DynDOLOD is "figured out".
  7. Haven't done in-depth testing, or anything below #7.

One thing I didn't mention in my previous list is that there are a couple of warts I'd like to clean up before packaging it up and distributing it widely:

  1. I'm not a fan of it creating individual *.md5 files to cache the MD5 checks.
    1. Maybe I can leave it halfway so that it'll use *.md5 files if they exist, but it won't create them.
  2. I did a short review of everything yesterday and noticed that I still have hardcoded numbers in the XML file that fix up the [MEMORY] section of enblocal.ini.
    1. I don't really want to try to bother detecting the "right" numbers, especially not right now, but maybe I should scale them back to something "safer" instead of just the numbers I use on my own system.
  3. The "STEP Optimized Textures" stuff is supposed to allow either the "Performance" or the "Standard" version, but the current version of my tool will only do the right thing for the "Performance" version, because the BSA / ESP files are named differently between the two.
    1. It's easy on my part to just say "you must use the Standard version", especially since that's Baseline.  I think I'll just do that.

So the new list of things to do before I package and distribute a ready-to-use version of this tool (in a "beta" state, of course):

  1. Do the right thing for #1, #2, and #3 in the list immediately above this one.
  2. Consider perhaps a slightly "nicer" version of the bashed patch creation.
  3. Test the output, to the best of my ability.
  4. One final pass through all the archives, in case someone released a newer version of something in the past month or so.
    1. If so, update and smoke test.
  5. Write some documentation about what it does and how to use it, aimed at novice and power users.

  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users