Jump to content


Photo

Release: StepperUpper


  • Please log in to reply
162 replies to this topic

#1 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 26 October 2016 - 04:47 PM

StepperUpper: STEP Up Your Game
A command-line utility to automate most of the tedious parts of setting up STEP, or other Skyrim packs like it as long as the steps are described precisely, with as little user interaction as reasonable.
STEP Core and STEP Extended are supported by the author, writing in third-person for some reason. Support for other packs is possible, with lots of manual effort to set up the pack definition files.

Latest Version: 1.1.0.0.
Latest Non-Beta Version: 1.0.0.0.

All release packages are signed with my OpenPGP key, 0E479CD4.

Latest Pack *.XML Files
These are maintained in the same GitHub repository as the software itself, in a special subfolder. I'll try to keep these up-to-date within a few days or so, and I'll especially try not to move them around now that there are links to them. No promises, though.

Right-click > Save As will work.
 
Current file version: 1.2.1.16, updated May 1, 2017.

*STEP Core and STEP Extended have a few "special" tasks that are particularly challenging to automate nicely, and which tend to take longer than everything else combined.  These "special" tasks for STEP Core are pretty much just related to DynDOLOD, and so it's somewhat reasonable to skip them, but for STEP Extended this also includes the DSR patch and FNIS generation.  For various reasons, these "special" tasks are separated out into their own "Finalize" scripts.  When you choose to run a process that includes "Finalize" scripts, be very careful not to click around on anything that pops up on the screen while it's going, or else you might accidentally make it hang it forever and need to start over.  It's probably best to just take your hands off the keyboard and mouse while it's going and just sit there.  I'm actually not even sure if it would be OK to lock your screen while this is going on.
**SR:LE still requires a lot of manual cleanup tasks after it's done. I've done a lot to help with these, but it's still really easy to screw up. For the time being, see the announcement post for what you need to do after this runs.  Edit: I'm no longer heavily maintaining this.  As time goes on without the original guide being maintained, more and more included pieces become unavailable, which makes this drift further and further away from a complete install.  It's too much work for me to just patch individual details if it's not going to be all that useful anyway.  I'm not going to outright delete the XML file, just know that it's basically not changing anymore.

In addition to these, I also keep some of my own tweaks in that "subfolder" link as separate files. Partly to demonstrate how you can make your own custom off-shoot of the process without needing to edit the official files, and partly because it's a convenient way for me to host these in the cloud so I don't lose them. Mostly the first thing, though. Definitely.
 
Demo
I ran through this kinda quickly, but here you go (as of version 0.9.3.1):
 

 
THIS. IS. BETA!!!
Any versions that I call "Beta" are not totally fleshed-out and bug-free. The only reason I'm releasing them at all is because I think it's probably still better for many users to have these awkward / klunky versions earlier.

Source Code
This is 100% open-source. Source code is hosted on GitHub and distributed under the MIT License.

Credits to third-party tools it uses can be found here.

Documentation
See the GitHub wiki for the project.  I think it doesn't suck, and I intend to keep it up-to-date as this evolves.

This is on that page too, but note specifically that you need .NET Framework 4.6 or higher to run this. 4.6 is included with Windows 10. Otherwise, if you don't have it already, you can get 4.6.2 from here.

Issues
I think I have a really good memory, but I still forget a few things sometimes, and I get really disorganized when left to my own devices. For both of those reasons, I use the Issues on the GitHub repository to try to mark down all the things I want to change about this thing going forward. You can throw stuff on there too, if you have a suggestion or if you think you found a bug.

I'd rather not use GitHub issues for technical support (the STEP forums seem like a better place to do that).

Changelog (Newest On Top)

  • v1.1.0.0: First beta release with support for SR:LE.
    • I've stopped putting pack files in with the release archive because especially in the SR:LE case, they get outdated really quickly and I want to encourage grabbing the latest versions.
    • A bit less special fancy code in the plugin cleaning process, which should make it a bit faster in some cases.
    • Support extracting the contents of BSA files (currently only hooked up to the code that runs after a mod archive has been extracted to the temp location).
    • Using something called Costura.Fody to clean up all the garbage in the release folder... almost all of the garbage is necessary one way or another, just Costura.Fody packs it all in with the .exe file at the cost of a slight startup-time hit.
    • Add a task to copy a MD5-checked file directly to the output directory without any processing beyond the initial check.
    • Add a task to just delete a file from the output directory.
    • Specializing some of the code for microarchitectures that don't need software to fix up unaligned memory reads... ELI5 version is that I made some of the custom plugin cleaning code a bit faster on computers that can run Skyrim in the first place.
  • v1.0.0.0: First non-beta release.
    • Minor tweak in the process... it looks like when it tried to report missing files when the output directory didn't exist yet, it would fail.
    • Some other minor tweaks in the process that you wouldn't notice unless you tried to run it while in early development for pack files supporting other packs.
    • Finally fixed that aMidianBorn Solstheim Landscape Mod ID so the link doesn't point you to the page for aMidianBorn Farmhouse.
  • v0.9.4.0 (Beta): Fifth, and perhaps final, beta release.
    • Isolated LOOT metadata so it's specific to this setup.
    • Pre-loaded the STEP LOOT user metadata rules.
    • Supporting SHA-512 checksums; added them to the things we download and run automatically.
      • Note: putting them on everything made things too much slower, so it's just the critical stuff.
  • Got rid of checks for the DLC BSA files since they can and do differ across languages (and it made things slower).
  • Setting Skyrim full-screen mode and resolution based on new inputs (defaulting to somewhat reasonable values).
  • v0.9.3.2 (Beta): Minor beta revision to 0.9.3.1 to fix #27.
  • v0.9.3.1 (Beta): Minor beta revision to 0.9.3.0 to fix #24.
  • v0.9.3.0 (Beta): Fourth beta release.  Many small changes.
    • Presenting a UI for the inputs if the download folder, pack files, and/or output folder are unspecified.
    • Allowing pack files to use the Skyrim install folder instead of Steam install folder for everything, and auto-detecting the Skyrim install folder if the Steam install folder is omitted.
    • Auto-detecting the 32-bit Java "bin" folder if it's omitted. Between this and the previous item, this fixes #17.
    • A bit of work towards #6 by adding a Game attribute to <Modpack> elements.
    • Optionally (enabled by default) pausing at the end of the process so users can see the output.
    • Running 7z.exe processes at the "Below Normal" priority so that system responsiveness doesn't completely tank while running this.
    • (in pack files) Wrye Bash now stores its settings in profile-specific folders instead of the default system-wide steamapps\common\Skyrim Mods. The Mod Organizer executable to run Wrye Bash has been changed to a batch script that makes this work.
    • (in pack files) Took out the 5-second delay when doing the DynDOLOD part of "Finalize" scripts, since it seems to work without it and it would have needed a WinActivate / WinWaitActive fanfare anyway.
    • (in pack files) Consistently using forward-slashes wherever possible instead of backslashes that occasionally need to be escaped and may make future cross-platform ideas more difficult.
  • v0.9.2.0 (Beta): Third beta release.  Changes are mainly to support path length pre-checks and enable running complex post-install steps using embedded AutoHotkey scripts.
    • <Modpack> elements now support an optional LongestOutputPathLength attribute which indicates the length of the longest sub-path under the output directory that needs to be supported for that pack.
      • When given, the application will refuse to run if the output directory length plus LongestOutputPathLength (plus one for the slash that combines them) exceeds 255, and so Windows will probably reject.
      • This checking behavior can be turned off with a new optional command-line flag, --allowLongPaths, in case you have reason to believe it's not an issue.
      • For pack developers, you can also pass a new optional command-line flag --detectMaxPath to have StepperUpper automatically detect the path string that would likely determine the correct value to put for LongestOutputPathLength (not guaranteed to be correct if the true determinant is very short-lived such that FileSystemWatcher doesn't have the opportunity to tell us about it).
  • Added a new type of task, <RunProcess>, which lets you run any arbitrary process.  Arguments can be treated as either bare strings or as relative paths rooted at the output directory.
    • There are now sample XML files that use this along with <Embedded> to run AutoHotkey scripts that automate DSR patching, FNIS output generation, DynDOLOD TexGen, and the main DynDOLOD generation.
  • Added various self-explanatory folder manipulation tasks: <DeleteFolder> and <MoveFolder>.
  • Added a new type of task, <EditFile>, which lets you make small changes to text files.  Currently supports adding a new line before or after another given line, modifying a line, and deleting a line altogether.
  • Fixed a bit of a race condition where consoles like the revamped Windows 10 one that deliver messages insanely slowly would make it look like the number of remaining tasks bounces around all over the place. I figured this wouldn't be as big of a deal as it was.
  • Changed the folder name format for the temporary "staging folders" that many archives extract to in order to let this succeed for deeper output folder paths than before, given the Windows path length limitations.
    • Along with this, some sample XML files switched from "SimpleMO" to more explicit and verbose equivalents to get even more room in the output folder path string.
  • v0.9.1.0 (Beta): Second beta release. Changes are mainly to support STEP Extended.
    • <Plugin> elements for <Clean> tasks no longer support their own individual WaitFor attributes.
      • Dropping it early makes sure that people don't get too attached to this.
      • Keeping this around would complicate #4, and in real-world use cases it will not offer a significant advantage over simply making the entire <Clean> task wait for everything that its plugins depend on.
    • -p now accepts multiple files to run in sequence. The long argument has been renamed to --packDefinitionFiles accordingly.
    • New (sometimes) optional argument: --javaBinFolder, used for filling in {JavaBinFolderForwardSlashes} placeholders in the pack XML files.
      • Required if any file in -p has {JavaBinFolderForwardSlashes} in it, otherwise optional.
      • Must be 32-bit to work with Mod Organizer. StepperUpper does not check for this.
    • <Modpack> elements now support a Requires attribute which allows them to specify other packs (optionally with the specific pack version, optionally with the specific file version) that must have installed successfully before running.
    • <Modpack> elements now must provide a MinimumToolVersion, starting this release.
      • Now, XML files intended for newer versions of the tool which are running on older versions of the tool that don't support all its features will cause an error right away instead of perhaps silently ignoring things it's asked to do.
    • <Hide> now appends a .mohidden extension instead of just deleting the file / folder entirely.
      • This allows the user to "Unhide" the hidden files through Mod Organizer if they choose.
    • Added <Optional> as a partial replacement for <Hide>, which does what Mod Organizer does when you move an ESP file to the "Optional ESPs" section.
      • Like the <Hide> change, this allows the user to manage these using Mod Organizer's Optional ESPs feature.
    • Sample files use <![CDATA[...]]> in embedded text files to make it easier to replace their contents.
    • Version updates for a few mods that have been updated in STEP Core.
      • I'm a bit confused about why ENBSeries v308 has changed since 0.9.0.0...
    • Providing a sample file for STEP Extended.
      • See #19 for manual steps you must do after the automation finishes, at least for now.
    • Missing files are now displayed in the user's web browser alongside links that go directly to the pages where the files can be downloaded.
  • v0.9.0.0 (Beta): Initial release.
    • Command-line application.
    • Scans a download directory (specified by the user) and the steamapps/common/Skyrim subdirectory of a Steam directory (also specified by the user) for available files.
    • Uses a *.xml file (specified by the user) to determine what files to look for and what tasks to run. Tasks include:
      • Copying files and creating new empty folders.
      • Extracting an archive using 7-zip (bundled), controlling exactly where the output goes.
      • Cleaning master and plugin files, given a list of records to delete, records to undelete and disable, and fields to delete from records.
        • Note that the resulting cleaned plugins will not be exact binary matches with what xEdit would produce, even though the content should be exactly the same.
    • Modifying or creating *.ini files.
    • Writing out embedded files verbatim, either text or base64-encoded.
    • Creates a near fully-installed pack as defined by the *.xml file.
      • DynDOLOD content generation is not done automatically since STEP Core is not necessarily your final stop, but we do set up everything you need to do so, including adding the executables to MO's list and extracting DynDOLOD Resources to the right place (leaving it disabled).
    • Bundled with a *.xml file that installs STEP Core 2.2.9.2 using the latest (at the time) versions of all the mods except for those that are explicitly left at their old versions (e.g., Skyrim Flora Overhaul).
    • Tries to make full use of all available hardware resources.
  • Thanks
    Thanks to all the people whose work I built upon to make this happen. Here are some of those people:
    • Bethesda for continuing to make their games in the Elder Scrolls series open to modding in the first place
    • The modders who collectively dedicated an incredible amount of time to building their own part of this cathedral
    • The STEP team, for obvious reasons too numerous to list
    • The contributors who put together the Tes5Mod category on UESP Wiki, which has been my "bible" of sorts while building that little automated plugin cleaning component
    • Forum users zilav and Kesta for their help clearing up some non-intuitive changes that xEdit makes to the Bethesda plugins
    • Contributors to LOOT, xEdit, Mod Organizer, and Wrye Bash, which I used to pre-cook parts of the XML file that users would otherwise have to do themselves
    • Everyone mentioned in the "Automated (Partial) Solutions" section of the Automation Survey wiki page, whose work inspired me to try something myself and showed me that there's a real desire for something like this to be done
    Original Post

Edited by airbreather, 01 May 2017 - 07:32 AM.

  • 2

#2 hishutup

hishutup

    Daedric Prince

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2,566 posts

Posted 26 October 2016 - 07:22 PM

First off, hey thats cool, now if I ever need to do something with a perfect step setup I can just click and go for the most part.

 

Theres a few things that I would like to see

  1. Toggling the cleaning of ESMs, or use something like bindiff to get the xEdit result.
  2. Marking files as optional if they do not contain a plugin or scripts that are required by the patches. I don't like the "optimized" things.
  3. Have a way to automatically open the pages so I can click the download button rather than having to copy/paste everything.
  4. Have an option to automatically extract BSA's when the plugin is empty which would allow me to not need MOs BSA management.
  5. Provide a mod with suggested and required INI Tweaks that can be toggled in MO, spINI used to provide this and I use this method today as I still have bad experiences with Bethlib not doing what I expect
  6. Instead of launching with args can you provide a UI.
  7. I think the directory of skyrim can be found automatically because MO can do it.

Theres more but they escape me.



#3 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 26 October 2016 - 09:48 PM

First off, hey thats cool, now if I ever need to do something with a perfect step setup I can just click and go for the most part.

 

Theres a few things that I would like to see

  1. Toggling the cleaning of ESMs, or use something like bindiff to get the xEdit result.
  2. Marking files as optional if they do not contain a plugin or scripts that are required by the patches. I don't like the "optimized" things.
  3. Have a way to automatically open the pages so I can click the download button rather than having to copy/paste everything.
  4. Have an option to automatically extract BSA's when the plugin is empty which would allow me to not need MOs BSA management.
  5. Provide a mod with suggested and required INI Tweaks that can be toggled in MO, spINI used to provide this and I use this method today as I still have bad experiences with Bethlib not doing what I expect
  6. Instead of launching with args can you provide a UI.
  7. I think the directory of skyrim can be found automatically because MO can do it.

Theres more but they escape me.

Thanks for the feedback, I really appreciate having another perspective on the things that a tool like this should do.  I have to admit I mostly spent my time thinking about what "common" / "regular" users would need, since that's what I considered myself when I started this journey (and maybe I'm still closer to that than I am to a "power user") and that's where I think something like this is most likely to have the greatest impact.

  1. Initially, my goal was in fact to match xEdit exactly.  It's technically still possible, just annoying (for me) and the remaining things I'd have to edit to achieve parity with xEdit just seem really unnecessary.
    1. For example, there are some cases where xEdit wipes the payload data for records that have been marked as deleted, and some cases where it leaves the payload data for deleted records alone, even within the same file.  I imagine that xEdit only wipes records in top groups it's already modifying anyway (this seems to be a theme with xEdit).  I don't really see a point in making special-case code, complicating the XML file, or both, just to make this part of the output file look exactly like what xEdit would have done in that case.  My optimization routine just wipes all the payload data for "deleted" records, which technically feels better to me than leaving them alone.
    2. Another example of a difference is sorting.  When xEdit deletes some records, it seems to reorder all the records in the same group as it.  It does a similar thing when it deletes groups (other than top groups).  I don't see a compelling reason to shuffle records or groups around like this.
    3. There are other examples too.
    4. I also thought about trying a binary diff tool of some sort really early on, but when I took a peek at the difference between "pre" and "post" cleaning, it was really crazy (which I now think was probably due to the sorting).  I think it'd be easier for me to just implement the individual tweaks to make this do the same things that xEdit does, especially now that this is out there and I don't have to feel like I'm withholding 100% of the program over a matter of less than 5% of the functionality... it's really more a matter of "why bother?".
  2. I'm not sure I understand this one.  Are you suggesting something like: "We noticed you don't have Fuzzy Bunny Lumpkins Mod 2.0.  This is marked as optional, so you can get away with skipping this.  Proceed anyway?  (Y/n)" and then skipping the tasks that use the Fuzzy Bunny Lumpkins Mod 2.0 file?  If so, I see the point, but I also see a couple of reasons not to add support for that kind of thing right away:
    1. If someone wants to also install a STEP Pack which implicitly relies on some objects added by Fuzzy Bunny Lumpkins Mod 2.0, the pack becomes more complicated than "you need to have STEP Core 2.2.9.2 installed to use this pack".  It has to instead be something like, "you need to have STEP Core 2.2.9.2 installed to use this pack, and you need to also have either not skipped over Fuzzy Bunny Lumpkins Mod 2.0 during the installation process, you need to provide it now, or you can skip over this dependent thing because it happens to not be necessary either (this last option might not be available all the time because the dependant might be a critical part of the pack)".  This makes "packs" either more complicated, more fragile, or both.  I'd rather have the XML files enforce a fairly rigid "baseline" that you as a user have to satisfy in order to get the tools to work, and then after you get your baseline set up is when we take the proverbial training wheels off and set you free to break everything customize to your heart's content.
    2. This makes it feel significantly more complicated, and there's a pretty easy workaround to get the exact behavior you want: just delete those parts of the XML file that do things you don't want to do.  Admittedly, editing the XML file isn't the most interesting thing in the world, but if it's just deleting stuff then it shouldn't be that bad.
  3. Yep.  Improving this part of the experience is very high on my to-do list for this project.
  4. I hadn't given any consideration to the idea of deeper BSA operations, but lordofla on /r/skyrimmods also had a similar comment right away after I posted over there, so there's definitely something to that idea.  I'll see what it takes to "teach" this about BSA stuff.  Note that I don't think it's a high-priority goal to make this happen right away, for reasons I mentioned on that link.
  5. I don't understand this one at all.  Can you please explain more about what you'd want the tool to do?
  6. I did have this thought a while ago, and I'm a bit torn on this.  On the one hand, it's clearly better to have a UI than not to have a UI, especially given that "regular" users are my target audience.  On the other hand, command-line args are really easy for me to do, and having them there allows me to spend more time helping with other things that are probably much harder for "regular" users than entering in command-line args.
  7. Oh, cool.  I'll look into that.  If it's easy, I'll switch that arg to "optional" and default it to the same auto-detection that MO does.

  • 0

#4 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,455 posts

Posted 27 October 2016 - 10:33 AM

I can't promise that I'll be able to test this any time soon, but I like what you're doing!



#5 Ganda

Ganda

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 375 posts

Posted 27 October 2016 - 10:38 AM

Hishy meant BethINI instead of Bethlib (hishy, shut up, jeez <.<)

 

As for the BSA stuff the easiest is to go with BSAOpt which is decently fast afaik and has a pretty great cli or just roll out your own lib in whatever language you're using.


  • 1

#6 Kesta

Kesta

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 468 posts

Posted 27 October 2016 - 10:49 AM

 

  1. Oh, cool.  I'll look into that.  If it's easy, I'll switch that arg to "optional" and default it to the same auto-detection that MO does.

 

You'll find it in the registry, the key should be under @"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\bethesda softworks\Skyrim", the value should be "installed path".

 

 

Edit: As for the .bsa, I'd suggest waiting. There have been discussion lately about a public archive lib handling old bsa, new bsa, and ba2, and Last time I heard Jon might come up with it packaged in the next update of BAE (which should handle the new SSE bsa format).


Edited by Kesta, 27 October 2016 - 10:51 AM.

  • 0

#7 Mator

Mator

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 463 posts

Posted 27 October 2016 - 04:52 PM

First off, hey thats cool, now if I ever need to do something with a perfect step setup I can just click and go for the most part.

 

Theres a few things that I would like to see

  1. Toggling the cleaning of ESMs, or use something like bindiff to get the xEdit result.
  2. Marking files as optional if they do not contain a plugin or scripts that are required by the patches. I don't like the "optimized" things.
  3. Have a way to automatically open the pages so I can click the download button rather than having to copy/paste everything.
  4. Have an option to automatically extract BSA's when the plugin is empty which would allow me to not need MOs BSA management.
  5. Provide a mod with suggested and required INI Tweaks that can be toggled in MO, spINI used to provide this and I use this method today as I still have bad experiences with Bethlib not doing what I expect
  6. Instead of launching with args can you provide a UI.
  7. I think the directory of skyrim can be found automatically because MO can do it.

Theres more but they escape me.

 

All of this is planned in the Mod Picker Mod List Setup Utility.
 

  1. Easiest way to do this is to just tap into the xEdit API directly.  https://www.reddit.c...port_9/d7tfy2x/
  2. A way to toggle the installation of individual mods in a mod list is something we plan to build.  That way if a user wants to download a mod list but not the CBBE nude females mod in it they can do that easily without having to clone the mod list on Mod Picker and then remove CBBE from it.
  3. We've been planning this as well.  It's not that hard to make an element in GUI which, when clicked, opens a link.  From the command line it's impossible though, to my knowledge.  You could have a prompt to open links though.  "Open 'https://....' ?  Y/n"
  4. This isn't something I thought of, but it could be fairly easily done.  There are tons of different BSA APIs out there, and I've already tapped into them before for Mod Analyzer, so it should be trivial to extract stuff.
  5. This is a good idea as a way to package INI changes.  Unfortunately, this can only apply to INIs that are directly managed by MO, and only to people who use MO to manage their mods.  I don't think it's difficult to do though.  Just create a folder in the MO mods folder, call it "INI Tweaks" and plop the INIs in there, ish.  I'm not 100% familiar with how this all works, but that's my understanding.  This would of course be something a user can toggle, so if a user wants to write directly to the profile's INIs they can do that.
  6. UI is a lot of work but an absolute must for an application like this to be usable by the general user.  We've been planning this since day 1.
  7. This can be done with the registry.  I've done it multiple times.  Here's some Delphi code that does it generically for all games with all possible registry keys (there are a number of different keys that could be set, depending on the version of the game): https://github.com/m...r.pas#L247-L287

  • 0

#8 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 27 October 2016 - 09:06 PM

All of this is planned in the Mod Picker Mod List Setup Utility.

  • Easiest way to do this is to just tap into the xEdit API directly. https://www.reddit.c...port_9/d7tfy2x/
  • A way to toggle the installation of individual mods in a mod list is something we plan to build. That way if a user wants to download a mod list but not the CBBE nude females mod in it they can do that easily without having to clone the mod list on Mod Picker and then remove CBBE from it.
  • We've been planning this as well. It's not that hard to make an element in GUI which, when clicked, opens a link. From the command line it's impossible though, to my knowledge. You could have a prompt to open links though. "Open 'https://....' ? Y/n"
  • This isn't something I thought of, but it could be fairly easily done. There are tons of different BSA APIs out there, and I've already tapped into them before for Mod Analyzer, so it should be trivial to extract stuff.
  • This is a good idea as a way to package INI changes. Unfortunately, this can only apply to INIs that are directly managed by MO, and only to people who use MO to manage their mods. I don't think it's difficult to do though. Just create a folder in the MO mods folder, call it "INI Tweaks" and plop the INIs in there, ish. I'm not 100% familiar with how this all works, but that's my understanding. This would of course be something a user can toggle, so if a user wants to write directly to the profile's INIs they can do that.
  • UI is a lot of work but an absolute must for an application like this to be usable by the general user. We've been planning this since day 1.
  • This can be done with the registry. I've done it multiple times. Here's some Delphi code that does it generically for all games with all possible registry keys (there are a number of different keys that could be set, depending on the version of the game): https://github.com/m...r.pas#L247-L287

For #2, would this just involve a mega database of "this mod uses features from that mod", "installing this mod in a pack that also includes that mod involves taking this special action", etc.?  I suppose that's probably what your Mod Picker site aims to achieve, but it's also a very large project and makes automation very tricky to get right.  I think this will be especially true in the "early" stages and for the less popular mods, where everything's wrong and users don't really know exactly how wrong it is until they commit a ton of time to testing it (which my gut says won't be nearly enough, and there will be much more frustration before there is relief).  I wish you all the best on your automated utility, and I'd be glad to help bounce ideas back and forth when your team sits down to seriously work on that, but I suspect that it won't completely replace the need for a "power tool" like StepperUpper, where a dedicated group of curators can pore over getting a small number of manually(ish) maintained XML files exactly right to minimize user error and support effort needed.
 
#3, yes there's an obvious way to do it with a GUI application, and it's a challenge to do the same idea with a command-line application, but I think you may be writing it off too quickly ::):.  The problem to be solved is not "allow the user to click a link on the command prompt", it's "make it easier for users to download the missing mods, given that the status quo in StepperUpper involves having to copy-paste URLs from the command prompt".  An alternative solution would be to just point their web browser at all the pages at once, or at a generated HTML file that also has the links along with what to grab from those pages.
 
#6, you're not wrong, but I do want to point out that just because my little tool today gets a failing grade on any reasonable "usability" score doesn't mean that I haven't also been "planning this since day 1", to some extent or another.  There are a few reasons why I pushed a version out without making a friendly interface yet, and why I will be OK with releasing a non-"beta" version even if it still requires command-line arguments (if it comes to that):
  • Better than nothing.  Especially for a project like STEP where there's a pretty well-established pack definition, a command-line application that takes arguments and a pre-cooked text file (even if it's a thing of nightmares to maintain the text file) is far better for the users than having essentially no automation support at all.  Users are not going to run away screaming just because it has command-line arguments.  They're probably already running away screaming because the next-best alternative is much worse.
    • The Mod Picker utility probably does make sense to have a bigger focus on usability, because it looks from a glance like you're aiming to make an ecosystem where pretty much anyone can make pretty much any kind of mod pack, pretty much anytime they want to, and Mod Picker will pretty much act as an AI to solve pretty much all of the problems that currently cause propositions like "anyone can be a curator!" to sound pretty much entirely unrealistic today.
  • TimingSSE will be available to the general public in... oh, it's available now.  That means a resurgence of Skyrim popularity, which means more people looking into modded Skyrim.  If the state-of-the-art of installing this stuff throughout this wave were still "here are the ~170 individual mod files you have to install and the instructions for the ~50 things that you have to manually fix up to get STEP Core to work... then you can move onto the actually complicated stuff in STEP Extended!", and I could have made a real difference but didn't because I wanted to hold out for a solution that's easier to use, then that's a textbook example of irony.  Also, I would be sad and probably regret not doing exactly what I'm doing now.
  • History.  Before I even lifted a finger to work on this tool, I looked at what it ought to do, and what those who had come before me had done / tried to do.  Conceptually, it seems really straightforward to automate this kind of stuff, and there are other modding communities that do have automated tools (CKAN for KSP, FTB for Minecraft, just to name the ones I've used myself).  Given that, it felt really far-fetched to imagine that out of the tens or hundreds of thousands of users who have gone through the process of setting up STEP alone (not to mention other mod packs that aren't STEP), nobody had seriously sat down and tried to automate any significant chunk of the process.  More likely, it seemed, there have been at least a few attempts (the word "Thunderbolt" vaguely stood out in my mind), but all of them fizzled out for one reason or another.
    • I suspected the main reason why nobody had gotten anything useful out to the public was because of an overabundance of ambition without having the ability to deliver on all of the goals ("eyes bigger than one's stomach", as the idiom goes).  I figured there were other factors, but trying to do too much and falling apart was what I identified as being the most likely way my own project would fail.  So I made a commitment to myself at the start not to be the 6th failure to release anything better for the users than manual steps, and to instead try to favor an MVP approach so that in the worst-case, there would be something out there that others could build off of, even if it objectively sucks.

You'll find it in the registry, the key should be under @"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\bethesda softworks\Skyrim", the value should be "installed path".


Edit: As for the .bsa, I'd suggest waiting. There have been discussion lately about a public archive lib handling old bsa, new bsa, and ba2, and Last time I heard Jon might come up with it packaged in the next update of BAE (which should handle the new SSE bsa format).

Thanks for the details.  I'll continue to wait on the bsa idea, though compared to the *.esm / *.esp manipulation I jumped on to do automated cleaning stuff, extracting contents of *.bsa files looks like a piece of cake to just do in C#.  Editing the files would have been a bit spookier, but probably not even all that bad either.

Edits: something weird happened with the quotes when I switched to the text-only markup editor... fixing that up.

Edited by airbreather, 27 October 2016 - 09:09 PM.

  • 0

#9 Mator

Mator

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 463 posts

Posted 28 October 2016 - 12:51 AM

For #2, would this just involve a mega database of "this mod uses features from that mod", "installing this mod in a pack that also includes that mod involves taking this special action", etc.? I suppose that's probably what your Mod Picker site aims to achieve, but it's also a very large project and makes automation very tricky to get right. I think this will be especially true in the "early" stages and for the less popular mods, where everything's wrong and users don't really know exactly how wrong it is until they commit a ton of time to testing it (which my gut says won't be nearly enough, and there will be much more frustration before there is relief).


Yes, the Mod Picker database does track Mod Requirements, along with plugin master dependencies, and a huge number of other datapoints. Each mod submitted to the database includes an analysis which is a JSON file produced by Mod Analyzer which includes the following:

  • A listing for every archive for the mod, including optional archives, with their filenames and filesizes.
  • A complete asset file map, including assets inside BSA archives.
  • Parsed FOMOD options, with asset maps defined per fomod option.
  • An analysis of every plugin in the mod, which includes the following metadata:
    • Filename
    • File size
    • CRC Hash
    • Masters (filenames, indexes, and CRC hashes)
    • Number of records
    • Number of override records
    • Authors Field
    • Description Field
    • Override records (signatures and FormIDs)
    • Errors (ITMs, ITPOs, UDRs, UESs, URRs, UERs)
    • Record Groups (Group Signature, Record Count, Override Count)

This data is then augmented by additional data scraped from sources where the mod can be downloaded, and input by the user. User input data includes:

  • Mod Requirements
  • Mod Categories
  • Mod Tags
  • Whether or not the mod is an external utility (installs files outside of the data folder)

Further data is submitted by users regarding plugin load order, mod install order, and mod compatibility, which allows us to help users build mod lists of compatible, properly ordered mods and plugins with full automatic resolution options.

The resulting mod list will then be able to be setup through our mod list setup utility.

 

I wish you all the best on your automated utility, and I'd be glad to help bounce ideas back and forth when your team sits down to seriously work on that, but I suspect that it won't completely replace the need for a "power tool" like StepperUpper, where a dedicated group of curators can pore over getting a small number of manually(ish) maintained XML files exactly right to minimize user error and support effort needed.


There's no reason why Mod Picker can't have a dedicated group of curators poring over the data associated with certain mods for a particular mod list to get everything right.

 

#3, yes there's an obvious way to do it with a GUI application, and it's a challenge to do the same idea with a command-line application, but I think you may be writing it off too quickly ::):. The problem to be solved is not "allow the user to click a link on the command prompt", it's "make it easier for users to download the missing mods, given that the status quo in StepperUpper involves having to copy-paste URLs from the command prompt". An alternative solution would be to just point their web browser at all the pages at once, or at a generated HTML file that also has the links along with what to grab from those pages.


Generated HTML file could certainly work. I personally like my idea of the prompt to open links thing. You could even do it in small batches (e.g. 5 or 10 mods at a time). Because, honestly, opening 100+ web browser links at the same time sounds like a terrible idea. XD

 

#6, you're not wrong, but I do want to point out that just because my little tool today gets a failing grade on any reasonable "usability" score doesn't mean that I haven't also been "planning this since day 1", to some extent or another. There are a few reasons why I pushed a version out without making a friendly interface yet, and why I will be OK with releasing a non-"beta" version even if it still requires command-line arguments (if it comes to that):

  • Better than nothing. Especially for a project like STEP where there's a pretty well-established pack definition, a command-line application that takes arguments and a pre-cooked text file (even if it's a thing of nightmares to maintain the text file) is far better for the users than having essentially no automation support at all. Users are not going to run away screaming just because it has command-line arguments. They're probably already running away screaming because the next-best alternative is much worse.
    • The Mod Picker utility probably does make sense to have a bigger focus on usability, because it looks from a glance like you're aiming to make an ecosystem where pretty much anyone can make pretty much any kind of mod pack, pretty much anytime they want to, and Mod Picker will pretty much act as an AI to solve pretty much all of the problems that currently cause propositions like "anyone can be a curator!" to sound pretty much entirely unrealistic today.
  • Timing. SSE will be available to the general public in... oh, it's available now. That means a resurgence of Skyrim popularity, which means more people looking into modded Skyrim. If the state-of-the-art of installing this stuff throughout this wave were still "here are the ~170 individual mod files you have to install and the instructions for the ~50 things that you have to manually fix up to get STEP Core to work... then you can move onto the actually complicated stuff in STEP Extended!", and I could have made a real difference but didn't because I wanted to hold out for a solution that's easier to use, then that's a textbook example of irony. Also, I would be sad and probably regret not doing exactly what I'm doing now.
  • History. Before I even lifted a finger to work on this tool, I looked at what it ought to do, and what those who had come before me had done / tried to do. Conceptually, it seems really straightforward to automate this kind of stuff, and there are other modding communities that do have automated tools (CKAN for KSP, FTB for Minecraft, just to name the ones I've used myself). Given that, it felt really far-fetched to imagine that out of the tens or hundreds of thousands of users who have gone through the process of setting up STEP alone (not to mention other mod packs that aren't STEP), nobody had seriously sat down and tried to automate any significant chunk of the process. More likely, it seemed, there have been at least a few attempts (the word "Thunderbolt" vaguely stood out in my mind), but all of them fizzled out for one reason or another.
    • I suspected the main reason why nobody had gotten anything useful out to the public was because of an overabundance of ambition without having the ability to deliver on all of the goals ("eyes bigger than one's stomach", as the idiom goes). I figured there were other factors, but trying to do too much and falling apart was what I identified as being the most likely way my own project would fail. So I made a commitment to myself at the start not to be the 6th failure to release anything better for the users than manual steps, and to instead try to favor an MVP approach so that in the worst-case, there would be something out there that others could build off of, even if it objectively sucks.

Sure, it's great! :)

I apologize if my message came of condescending in any way, that was not my intent. I will admit I am a bit frustrated that you completed this literally weeks before our similar solution was built, but it can't be helped. And, ultimately, having solutions competing with each other in the community can only be a good thing, as it inspires innovation and new ideas.

 

Thanks for the details. I'll continue to wait on the bsa idea, though compared to the *.esm / *.esp manipulation I jumped on to do automated cleaning stuff, extracting contents of *.bsa files looks like a piece of cake to just do in C#. Editing the files would have been a bit spookier, but probably not even all that bad either.

Edits: something weird happened with the quotes when I switched to the text-only markup editor... fixing that up.


Regarding BSAs, I urge you to check out mod-analyzer on GitHub. I did the work to port (most of) libbsa to a .NET interface so I could access it from Mod Analyzer to extract BSA archives, and I'm using BAE for BA2 archives. Because Mod Analyzer is a C# application you should be able to call these libraries with the same code from your application. Although I only needed getting the paths from the BSA/BA2 archive, so it'll be slightly different I suppose.


- Mator


  • 0

#10 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 28 October 2016 - 07:28 AM

 

I wish you all the best on your automated utility, and I'd be glad to help bounce ideas back and forth when your team sits down to seriously work on that, but I suspect that it won't completely replace the need for a "power tool" like StepperUpper, where a dedicated group of curators can pore over getting a small number of manually(ish) maintained XML files exactly right to minimize user error and support effort needed.


There's no reason why Mod Picker can't have a dedicated group of curators poring over the data associated with certain mods for a particular mod list to get everything right.

 

Sorry, what I meant by this was that my gut feeling is that there's always (or at least for a very long time) going to be some amount of edge cases in complicated loadouts that would be really straightforward to support with a "power tool" like StepperUpper, but really hard and awkward to squeeze into a "one size fits all" tool like the Mod Picker idea.

Maybe instead of "power tool", I should have said something more like "precision-guided missile", since that's a better metaphor for what I feel makes this different from the Mod Picker idea.
 

I apologize if my message came of condescending in any way, that was not my intent. I will admit I am a bit frustrated that you completed this literally weeks before our similar solution was built, but it can't be helped. And, ultimately, having solutions competing with each other in the community can only be a good thing, as it inspires innovation and new ideas.

I may be a young 27-year-old whipper-snapper, but I've been around on the internet long enough to know what to expect when I'm an outsider coming to a mature community proposing a radical solution to what I perceive to be their problems :;):.  #6 about the UI did totally come off as condescending.  That's fine with me.  It's even more fine since you said you didn't mean it that way.  I care a lot more about the content than the tone, and the point was worthy of a response.
 

Regarding BSAs, I urge you to check out mod-analyzer on GitHub. I did the work to port (most of) libbsa to a .NET interface so I could access it from Mod Analyzer to extract BSA archives, and I'm using BAE for BA2 archives. Because Mod Analyzer is a C# application you should be able to call these libraries with the same code from your application. Although I only needed getting the paths from the BSA/BA2 archive, so it'll be slightly different I suppose.

But... using what someone else wrote wouldn't be fun!1!11!!!!
 
More seriously, I'm not likely to jump on anything BSA-related for a little while here.  If I did, I'd probably try BSAopt first if it doesn't suck to drive it headless-style. Otherwise, I appreciate the idea, but I'd prefer trying to P/Invoke libbsa directly before jumping on a solution that hammers C++ source code into a shape that runs on the CLR (C++/CLI).  My point about writing another BSA reader in C# was just to point out that it looks really easy compared to the other stuff I've done here.

 
In other news, yesterday I got some support into the codebase for something like "packs" and got an XML file that tacks STEP Extended onto STEP Core.  I cheated a little by having STEP Core do some of the work since it already has the data around at the time.  For example, in the modified STEP Core XML file, SMIM is now three parts: Common, Jewelry Rings (Vanilla), and Jewelry Rings (CCO Remade) so that STEP Extended only needs to swap those two checkboxes.  Still not 100% sure what I should do about making the automated DSR patch and/or running FNIS, but... meh.  I'm fine with just leaving them as "post-automation" manual steps, especially for now.


  • 0

#11 Venom

Venom

    Prisoner

  • Members
  • 8 posts

Posted 30 October 2016 - 09:18 PM

I've been having a tinker about with this for the last couple of days and have a few thoughts and reports I wanted to pass on.

 

The first problem I hit was getting a lot of "Access Denied" errors after the unpacking phase that caused the program to crash out. I have since found out that this was due to some of the temporary paths exceeding the 256 character limit in Windows. I can't vouch for earlier versions of Windows, but in Windows 10 it is possible to remove this limit by going into Group Policy Editor (gpedit.msc) and enabling the following option;

 

Local Computer Policy --> Computer Configuration --> Administrative Templates --> System --> Filesystem --> Enable Win32 long paths

 

I don't know if a reboot is required afterwards, but it is worth doing when changing system related things anyway. It may be worth mentioning this so that users know it is an easy fix, and not a program error.

 

The next thing I ran into was getting a bunch of stack errors and program errors during the later stages of the program run. It appeared to happen at the point where it is tidying up the temporary folders, but I can't be certain. To make sure I was working from the latest code I compiled a version from the most recent GitHub code. This compiled fine and the program executed completely. I am unsure what caused the original error, but something has changed between the first release (V0.9.0.0) and the latest compiled version (V0.9.1.0) to remedy these errors. It may be worth updating the released version in case this is happening to anyone else.

 

Following all this, I now have a perfectly working install of STEP Core. As I have already made the preliminary manual actions, and downloaded all the required files to a safe download location, I can re-run the program (with the wipe option) and get a completely clean install within ten minutes. All that I need to do afterwards myself is update DynDOLOD and I am ready to go. A vast improvement over how long it used to take me do all this manually.

 

The next thing I have been playing with is trying to follow on with the STEP Extended install. I have been having some problems with this as it says it needs to be set up in the same run as STEP Core, but I can't seem to be able to pass both XML files in a single run. I have tried doubling up on the -p parameter, and merging the XML files myself, but with no success. I notice in your last post that you said you were just getting this working. Is it in any state to be released as I wouldn't mind having a play around with it. I would be happy to report back.

 

I apologise if any of this sounds like I am setting out to break things, but as I said, I have been tinkering. Even though I have a good degree of technical knowledge, I am trying to come at this as a user so that I can provide feedback of the experience. Up to now, I think this is a work of genius. It will make it so much easier for me to try out things without having to worry about breaking my install.

 

I think there could be a lot of demand for this as Skyrim Special Edition is probably not going to have a mature enough modding setup for a while yet. Although it looks beautiful once you are in the game, not having SKSE is a deal breaker for me due to the mods that depend on it. Hopefully, a compatible version of SKSE will be out before the end of the year. Until then, I am more than happy to play a fully STEPped version of the original.

 

I hope you manage to get something useful out of my inane ramblings. I am still playing around with it, and will let you know anything else I find.

 

Cheers,

 

Pete


  • 1

#12 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 31 October 2016 - 06:19 AM

I've been having a tinker about with this for the last couple of days and have a few thoughts and reports I wanted to pass on.

 

The first problem I hit was getting a lot of "Access Denied" errors after the unpacking phase that caused the program to crash out. I have since found out that this was due to some of the temporary paths exceeding the 256 character limit in Windows. I can't vouch for earlier versions of Windows, but in Windows 10 it is possible to remove this limit by going into Group Policy Editor (gpedit.msc) and enabling the following option;

 

Local Computer Policy --> Computer Configuration --> Administrative Templates --> System --> Filesystem --> Enable Win32 long paths

 

I don't know if a reboot is required afterwards, but it is worth doing when changing system related things anyway. It may be worth mentioning this so that users know it is an easy fix, and not a program error.

 

The next thing I ran into was getting a bunch of stack errors and program errors during the later stages of the program run. It appeared to happen at the point where it is tidying up the temporary folders, but I can't be certain. To make sure I was working from the latest code I compiled a version from the most recent GitHub code. This compiled fine and the program executed completely. I am unsure what caused the original error, but something has changed between the first release (V0.9.0.0) and the latest compiled version (V0.9.1.0) to remedy these errors. It may be worth updating the released version in case this is happening to anyone else.

 

Following all this, I now have a perfectly working install of STEP Core. As I have already made the preliminary manual actions, and downloaded all the required files to a safe download location, I can re-run the program (with the wipe option) and get a completely clean install within ten minutes. All that I need to do afterwards myself is update DynDOLOD and I am ready to go. A vast improvement over how long it used to take me do all this manually.

 

The next thing I have been playing with is trying to follow on with the STEP Extended install. I have been having some problems with this as it says it needs to be set up in the same run as STEP Core, but I can't seem to be able to pass both XML files in a single run. I have tried doubling up on the -p parameter, and merging the XML files myself, but with no success. I notice in your last post that you said you were just getting this working. Is it in any state to be released as I wouldn't mind having a play around with it. I would be happy to report back.

 

I apologise if any of this sounds like I am setting out to break things, but as I said, I have been tinkering. Even though I have a good degree of technical knowledge, I am trying to come at this as a user so that I can provide feedback of the experience. Up to now, I think this is a work of genius. It will make it so much easier for me to try out things without having to worry about breaking my install.

 

I think there could be a lot of demand for this as Skyrim Special Edition is probably not going to have a mature enough modding setup for a while yet. Although it looks beautiful once you are in the game, not having SKSE is a deal breaker for me due to the mods that depend on it. Hopefully, a compatible version of SKSE will be out before the end of the year. Until then, I am more than happy to play a fully STEPped version of the original.

 

I hope you manage to get something useful out of my inane ramblings. I am still playing around with it, and will let you know anything else I find.

 

Cheers,

 

Pete

First, don't worry about sounding like you're setting out to break things.  That's very much appreciated, and it's exactly the kind of stuff I was hoping to see when I called for testers in the initial post.  As long as it's not just trying to put everything into system32 or reading from a malformed XML file or something silly like that.

 

To tack on more packs with the latest thing from source when you downloaded it, it's -p "FirstFile|SecondFile|ThirdFile".  In case you know how to read C#, the relevant code for this was here and here... that Tokenize method is a bit overly "clever" compared to a simple String.Split, but it does the job.  In any case, since I started typing this, I've just committed a quick update that makes it so you can just do -p "FirstFile" "SecondFile" "ThirdFile".  I don't really know what I'm waiting for to push out a build with working STEP Extended.  Maybe I'll do that shortly after this post unless I find an issue.

 

So to clarify, when you posted your reply, it would have had to look like this:

StepperUpper -p "C:\Sample\STEP Core 2.2.9.2.xml|C:\Sample\STEP Extended 2.2.9.2.xml" -o "C:\Sample\Output" -d "C:\Sample\Downloads" -s "C:\Sample\Steam"

Using the latest pre-release version on GitHub as of the time I'm typing this message, it now has to look like this:

StepperUpper -p "C:\Sample\STEP Core 2.2.9.2.xml" "C:\Sample\STEP Core 2.2.9.2.xml" -o "C:\Sample\Output" -d "C:\Sample\Downloads" -s "C:\Sample\Steam"

The main advantage of this new way over the other way is that this way works much nicer when you drag-and-drop the files onto the command prompt to get the paths into there.  Another advantage is that you won't accidentally try to pipe the command output into a shell-opened file if you leave out the quotes.

 

As for Win32 long paths, I'd vaguely heard of that feature, but it looks like you heard correctly that this is only supported in Windows 10.  I'm not sure why that would work by itself for a .NET application, even if the 64-bit version of 7-zip happens to work OK with it... I'm guessing that you also changed StepperUpper.exe.config StepperUpper.exe.config to have it target 4.6.2 and/or disable the switches in the AppContextSwitchOverrides, and that's the part that actually made the rest of this work?

 

Also, the temporary paths look a bit long, but "Staging_ModName_abcdefgh.xyz" is only a few characters longer than "ModOrganizer\mods\ModName" which everything goes into.  So, if the temporary paths are too long, then the final paths will be too long too unless the output path length is in a really narrow range.  I guess I could shorten that a bit so that it's not longer, but that wouldn't fix it completely by itself because there are still situations where paths in the staging folders are longer because of "00 Core" (and worse) in archives with FOMODs.  "Real Wood Textures - Farmhouses - No Green Moss Version 1k (S.T.E.P. Version) 1.0" is the longest folder name that's at least partly my fault.  I always use -o "C:\Games\STEPDump" on Windows 7, and it's been fine for me so far.  I'll check to see what the longest paths are and if there's any easy way to optimize some of the names in the XML files so it's less likely to happen.

 

In any case, you say the length thing is not a program error, but I logged a bug for it anyway.  We wind up with some pretty long chains of files sometimes (even a highly optimized "O\mods\d\Meshes\dlc01\lod\castle\dlc01casextfrontrighttower01glow_lod_1.nif" takes up close to 1/3 of the available path length), and a goal of this is to do checks upfront where possible to alert you if something will fail later (thus the MD5 thing).  I'm also guessing that there are some "-o" lengths that will make StepperUpper chomp up your CPU and run forever because some of the code just immediately retries indefinitely on an error (which is not as crazy as it sounds because of the fact that we spin up over a hundred "7z.exe" processes at the same time, and so everything's going to be running slowly for a while).

Furthermore, I'm a little worried about going all-in on either enabling Long Paths on Windows 10, using the .NET 4.6.2 flag to enable probably a \\?\ hack if Win32 long paths are not available, or both, as opposed to just terminating with an error.  Will Skyrim + Mod Organizer do the right thing if it tries to load a file with a really long path name?  What about external tool processing like SkyProc patchers?  Mod Organizer, perhaps when you try to .mohidden a file?

 

One unfortunate part of the current process's workflow is that if there's an error that happens when running any individual task, you won't hear about it until all other tasks terminate (perhaps with their own errors as well), and I think (technical .NET jargon) an "await" on a "Task.WhenAll" will only rewrap one of the individual tasks' exceptions on a fault, so we might as well cancel everything as soon as anything fails.  That said, I honestly don't know what could have changed between the released version and a newly compiled 0.9.1.0 version to make it look like that fixed anything.

 

I'm glad you were able to get it to run and that you're able to benefit from it.  Hopefully in time, we'll be able to iron out the kinks to make it so more people with less patience can use it too ::):.


  • 0

#13 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 31 October 2016 - 07:12 AM

I don't really know what I'm waiting for to push out a build with working STEP Extended.  Maybe I'll do that shortly after this post unless I find an issue.

I remembered why I decided to sit on it for now (though if you hadn't brought it up, I probably wouldn't have remembered, and thus would have sat on it for longer...).

 

I really don't like this, and since you need to go through the DSR Patch and/or FNIS stuff in order for it to run properly (STEP Core can get away with you not doing DynDOLOD stuff), I was going to wait until I had a better solution for post-process cleanups.  STEP Extended more than triples the amount of time this takes on my computer because of all the manual steps plus DynDOLOD, and about half of the resulting process still needs babysitting.


  • 0

#14 Venom

Venom

    Prisoner

  • Members
  • 8 posts

Posted 31 October 2016 - 08:21 AM

I picked up on the updated code this morning and noticed the change you mentioned. I have now got it running all four packs in a single run by putting a space between them. I have got myself a nice updated batch file to kick off the process. Thanks for that.

 

The long paths became an issue for me as I am using the output path of "D:\Games\_Modding\Skyrim\STEP", so I am already 29 characters into my limit. The cause of the issue became apparent when I changed the output folder to "D:\X" and everything worked. You are probably right to create an issue for it as I can't find any viable workarounds for Windows 7 or Windows 8. There are mentions of a MAX_PATH setting within the Windows API, but no way of changing it. I also found that you can get around it by mounting a folder as a drive by using the subst command, but this starts to get tricky and very system specific. It kind of breaks the functionality of a generic tool. Keeping your output path short may be the best bet at the moment unless you are using Windows 10.

 

I agree with your point about the \\?\ hack as well. It should work fine with .NET applications, but once you start drawing in more third party applications, you start to be unable to predict how they are going to handle it. Your idea of a path length check at the beginning, would probably be the easiest and safest. You can then drop out with an appropriate error message.

 

I have deliberately tried not to change anything in the code myself to get it to work, so I didn't change any of the config values or switches. I just compiled the latest version and the stack errors stopped happening. It may be that compiling it on my system has kind of tailored it for my system. It would be worth getting input from other people who have used it to see what their experiences of this are as well.

 

Thanks for the list of post-process cleanup tasks. I was trying to put together something like that myself, but had missed out a number of things in your list. I can see why you are reluctant to release while these are still in place. It all seems to be application related, principally MO, so an AutoIT script should be able to take care of most of it. My knowledge of AutoIT is fairly limited though, so I'm not going to be a great deal of help there I'm afraid.

 

One other thing worth mentioning is that when specifying the javaBinFolder parameter, it needs to be to a 32-bit version of javaw.exe e.g. within Program Files (x86), as MO does not allow 64-bit applications to be executed.

 

I have also made my own copy of My STEP Core Tweaks.xml, and updated it to use my own ini changes, including within enblocal.ini. This has the benefit of removing the original step 6 (ENBoost memory config) as this can now be hard-coded in the xml file. This gives scope for additional user configuration, as I am sure we all have our own favourite ini tweaks. I'm sure most of us round here have experience fiddling around with the ini files and know how to avoid breaking them, but if this tool is going to try to bring new users in, it needs to be a bit more user friendly. It might be worth thinking about a front-end to create a user specific xml file to add on after Core and/or Extended. This could well lead on to a front-end to manage the whole process. These are merely thoughts for future reference though. I am a bit of a command line junkie, and appreciate the control that it gives me.

 

Once again, I seem to have rambled on. I should really get my thoughts ordered before writing things down, but that seems to be the way my brain works ;)


  • 0

#15 airbreather

airbreather

    Thane

  • Developers
  • PipPipPip
  • 101 posts

Posted 31 October 2016 - 08:52 AM

Released Version 0.9.1.0 anyway.  See the link for what's new.

 

I figure that even with the list of 34 or so manual steps, those same 34 or so manual steps would still be done as part of a manual STEP Extended install, plus a bunch more that are just as bad as this (or worse).  At least they're documented...

 

I did mention over on #10 that a GUI for managing XML files might actually be more worthwhile than a GUI for running the process itself, since the process itself is "fire and forget" whereas managing the pack definitions is a lot more interactive.

 

Yeah, --javaBinFolder does need to be 32-bit.  I don't currently know how to check for that.  Maybe I could look into redistributing / downloading a JRE just to make sure it's not left up to "chance".

 

If the final output paths wouldn't actually be bad and it's just the intermediate stuff, then maybe longer path support might be appropriate.  Or maybe just finding the correct combination of fancy 7z.exe flags to extract exactly what's needed to exactly the right place would be better, since that's something I've thought of a couple times now.


  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users