Jump to content


Photo

Catalog of Existing Solutions | Requirements for Any Tool


  • Please log in to reply
18 replies to this topic

#1 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 31 August 2016 - 08:50 AM

Background:

 

STEP is a pretty complicated thing to set up and configure.  Well, I'm a software developer ("senior software developer", whatever that means to you guys... titles, yay!) working on a desktop application that's around the same level of complexity in terms of setting it up and configuring it.  It's at least in the same ballpark; it's hard to say which is more complicated.  Our current work is to improve the experience of setup and configuration.  I saw the correlation with STEP and wanted to poke my head in to see if I could help out, because I know there are some pretty painful parts of the STEP process that can be done better.

 

OK.  That's who I am and where I'm coming from.  Now I'm going to go on to list what I see as the main tasks that need to happen to set up STEP Core (not in the order they need to happen):

  1. Install Skyrim Legendary Edition from Steam.
  2. Tweak Skyrim INI files.
  3. Set graphics driver settings.
  4. Download pre-packaged archive files from locations that are not friendly to unattended downloaders.
  5. Extract a subset of the contents of each archive file to directories in a well-defined directory structure (Mod Organizer's "mods" directory).
  6. Set up a Mod Organizer profile to order the mods in a particular way.
  7. Install other tools that don't sit in the virtual "Data" directory (SKSE, Mod Organizer, Wrye Bash, LOOT, TES5Edit) (I might have missed some more because it's been a while).
  8. Run TES5Edit through Mod Organizer and perform various script tasks (cleaning, DynDOLOD, other???).
  9. Run Wrye Bash and create a Bashed Patch.
  10. Run LOOT to set up the load order.
  11. Run various other tasks provided by the mods themselves, usually via SkyProc.

Does that look about right?  Did I miss anything big?  If I missed something, that should be addressed first.

 

Assuming I got the list of tasks right, only a subset of these steps need to be run during every upgrade: 4, 5, 6, 8, 9, 10, 11.

 

OK, now I have two questions:

  1. What steps are already covered (by one or more existing solutions in this forum?
    • I do know how to search, but I don't have the time to read through everything.  Plus, I think there's value in having the information in one place anyway.
    • It's fine if they're only partial solutions, or solutions to only parts of a task, or if it's broken right now, but it would be nice to know that.
  2. What steps are the most painful for users setting up a system that includes STEP?
    • I count setting up STEP Core from scratch, upgrading from an old version, and adding things on top of STEP Core as in-scope here.

I'd like to volunteer to at least keep a post somewhat up-to-date about what we've identified the problems to be, and what automated solutions exist for solving those problems.  I might even poke my head in and work on tools for some of this, but no promises there.

 

In my dream world, I (as a user) can grab a tiny little file that someone put together describing what a complete STEP Core install looks like, point a tool at that file and a folder of downloaded Nexus archives, have it tell me which files I should re-download and which files are missing entirely, re-download the missing / wrong files, and then have it do as much of the remaining setup tasks for me (file moves, extractions, running patchers, load order management) automatically that it reasonably and legally can, and go to bed (or lunch) while it does all that.  Then I'd do the things that still need to be manual, and then run Skyrim once that's done.  Still in fantasy land, I'd like to be able to do the same to install (and maybe even uninstall!) other packs that build off of STEP Core the same way.  Still in that fantasy land, it would be really really nice if this tool would look at Mod Organizer folders and detect work it doesn't need to do because it's already done (i.e., compare checksums on the installed files or something).

 

I think this isn't too far away from a possible reality, and I also think there's a lot of stuff in this forum that can be leveraged to make this work.

 

Tried to reserve a second post for a catalog that I'm going to try to keep up-to-date, when I get the time to do so, but apparently that just appends the text to the end of this top post?


Edited by airbreather, 31 August 2016 - 08:50 AM.

  • 0

#2 DoubleYou

DoubleYou

    Wiki Stepper

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,466 posts

Posted 31 August 2016 - 10:52 AM

I have actually been thinking about this as of late. Below is about as far as I got.

A tool that could scan a MO profile for all active mods, from the meta.ini files find the file names for those mods, grab the checksum of those files from the download directory, and put that in an editable file would be useful. The workflow would be best to only install one file into any mod folder (no merging) so that it can be fully automated with no manual editing. The tool could then scan a download directory for the required files, and if any are missing, provide a list with a "visit mod page to download file" button to launch the Nexus page from the Nexus ID.

The goal would be to allow pack authors to write their pack almost with the click of a button.

#3 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 02 September 2016 - 07:03 AM

I have actually been thinking about this as of late. Below is about as far as I got.

A tool that could scan a MO profile for all active mods, from the meta.ini files find the file names for those mods, grab the checksum of those files from the download directory, and put that in an editable file would be useful. The workflow would be best to only install one file into any mod folder (no merging) so that it can be fully automated with no manual editing. The tool could then scan a download directory for the required files, and if any are missing, provide a list with a "visit mod page to download file" button to launch the Nexus page from the Nexus ID.

The goal would be to allow pack authors to write their pack almost with the click of a button.

Hmm, automated pack creation does sound really useful.  One potentially big hiccup I see with that:

  1. I expect that most packs will require not only the same mod archive file(s) as Core, but a different subset of the data that's in the archive file itself.  At best, this greatly complicates the solution: in order for a meta.ini scanner to be perfect, it would have to correlate subtrees on-disk with sub-trees in the archive file.  More realistically, I expect this would look more like a message to the user saying "this was done all fancy-custom-like, so you'll have to do it manually".  I also expect this to be common.

 

In general, I'm not too concerned with making authors' lives easy, at least not as a primary goal.  What I would like to see is something like this:

  1. Installing an existing STEP Core / STEP pack is as easy and as automated as it can be for users.
  2. This could mean more users, which means more people who might want to try their hand at authoring a new pack.
  3. New authors spend most of their time with the "business side" of the problem: finding the mods to insert, testing compatibility, etc.
  4. Once it's done and tested, they take an example file from a different pack (or a template, maybe) and use that to guide them in making their pack available.

It sounds like what you're saying would just simplify that last step of the process.  I know it would suck if that step is too hard or complicated, but I also think it's likely to be far simpler and easier than the step that comes before it.

 

I'd love to be wrong, though.  I'd be very happy if we could make every other step of the process so comparatively easy that generating the description for a pack will in fact be a noticeably painful part of the process.


  • 0

#4 DoubleYou

DoubleYou

    Wiki Stepper

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,466 posts

Posted 02 September 2016 - 09:25 AM

I guess I just like the idea of automating the automation. My thought would be for it to work on any profile that a mod author could put together, altogether making it easy to maintain, as I imagine the greatest danger is with updates, keeping the beast maintained could pose some difficulty.

#5 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 04 September 2016 - 03:39 PM

OK.  I've discovered this SASTEP wiki page, so I'll get to cataloging what I can from that and other forum posts.  I have some free time today and a bit more tomorrow too.

 

Side note, can I not edit these posts after a certain period of time?  If that's the case, I can see why that one SASTEP page is on the wiki.


Edited by airbreather, 04 September 2016 - 03:40 PM.

  • 0

#6 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 04 September 2016 - 06:20 PM

OK.  I've discovered this SASTEP wiki page, so I'll get to cataloging what I can from that and other forum posts.  I have some free time today and a bit more tomorrow too.

 

Side note, can I not edit these posts after a certain period of time?  If that's the case, I can see why that one SASTEP page is on the wiki.

Yep I can't even edit the quoted post anymore.  To avoid spamming this forum, I'm going to stop posting here for regular run-of-the-mill progress and just mark down what I can come up with over on this Automation Survey page I've created on the wiki.  It's similar to "SASTEP", but different in that the goal is to start from a 1,000-foot overview of the entire STEP setup process and then address how automation can / does / should / might / shouldn't improve certain parts of it, and why not.  It looks like that other page started off with a goal similar to mine, but it died due to lack of organization / interest / engagement / whatever, as it hasn't been updated since December 2013 and a lot of links have since broken.

 

Perhaps this new page will spark ideas about how to automate parts of the process that nobody's even thought about yet.  Maybe it'll instead die tragically and count itself as another among what looks like a graveyard of half-baked ideas.

 

My overall meta-plan is:

  1. Broadly discuss the STEP installation process, with a focus on how automation might be able to help (and importantly, how it cannot and why).
  2. Identify what tools people have already written.
  3. Create some crappy little versions of "user stories" that clearly lay out (my understanding of) what users might be looking for.
  4. (maybe) Write some automation stuff myself, possibly looking into using Cake.

Everything starts off from an end user's perspective.  Pack developers can suffer, for all I care (kidding... I care, I just want to focus on the end users who will collectively install a single pack dozens / hundreds / thousands? of times over the course of months instead of a pack developer who's going to maybe spend a day or two setting up the automation for their pack and then ship it).


  • 0

#7 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 11 September 2016 - 02:33 PM

Status report time.  I've got nearly everything for steps 1 and 2 of my meta-plan onto that page.  There's quite a graveyard here of great-sounding ideas that never took off for one reason or the other.

 

I haven't quite read through the dozen pages of posts in the ASI thread yet, but from the looks of it, that could have been a promising start to a full solution.  Leaving out MO, however, makes it a non-starter for me.

 

As I had suspected, this forum has had a good few half-baked partial solutions to the "automate STEP" problem, but they all seem to fizzle out due to probably a lack of motivation / vision or having goals that are both attainable and useful.

 

With Skyrim Special Edition coming up soon and the STEP team planning to target STEP:SSE, I foresee there being a renewed interest in STEP or a similar project.  So I'm happy to help.

 

My immediate next goal is to start working on some crappy "user stories" to get some concrete targets.  Ideally, these targets would be realistic and reasonably useful.


Edited by airbreather, 11 September 2016 - 02:33 PM.

  • 0

#8 hishutup

hishutup

    Daedric Prince

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2,572 posts

Posted 11 September 2016 - 06:01 PM

I think the problem stemmed from that the nexus doesnt like hotlinking to files.

There is another project called skyrim mod picker which is pretty close to being finished.



#9 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 12 September 2016 - 05:30 AM

I think the problem stemmed from that the nexus doesnt like hotlinking to files.

Some of the attempted solutions (SASTEP, ASI, Mod Combiner) didn't include file downloads in the list of things they do.

 

There is another project called skyrim mod picker which is pretty close to being finished.

Looks like when that's ready, it's going to be a website that makes it easier to make and/or share packs.  If anything, it would help reduce the STEP developers' workload.  It doesn't appear to replace any of the need for a tool like this, especially since they've admitted that you still won't be able to download files directly from Nexus.


  • 0

#10 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 19 September 2016 - 06:22 AM

So, some progress has been made.  As I wrote in the wiki, I've now got a machine-readable XML file that, so far, just defines the list of files that the user needs to have in order for the setup process to do its job.  Probably unexpectedly, this also includes descriptions of the official DLC files so we can make sure that the user has the appropriate DLC (presenting Steam links to help if they don't) and to check to see if their ESM files are untouched, properly cleaned (or properly partially cleaned, in the case of Dawnguard.esm), or some other thing that would require the user to start over.

 

I also hacked together a command-line C# program (depends on Command Line Parse Library 1.9.71.2 stable (NuGet package)) that takes the XML file and scans a given download directory and the steamapps\common\Skyrim\Data subdirectory of a given Steam directory for all the listed files and collects the list of missing files into one array that we could do... something with later.

 

There's definitely going to have to be something to help maintain this XML file at some point.  Literally the morning after I wrapped up that XML file, USLEEP released 3.0.6 and removed the version I had in the XML file, so it needed to be updated.


  • 0

#11 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 23 September 2016 - 06:10 AM

More progress.

 

I've put all the work products on GitHub.  It's still kinda early, so it's not "nice" yet.

 

It depends on finding my AirBreather.Common repo in a specific place, as well as probably too many NuGet packages that I just haven't cleaned up yet.

 

The tool is now capable of INI file tweaks and extracting archives using 7-zip and moving subsets of the contents to the output folder.

 

My next goal is to flesh out the XML file with the description of which files in which archives need to be extracted where.  At the time of writing, it just does what's needed in "1.C. Install Utilities" and "2.C. Extenders".


  • 0

#12 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 27 September 2016 - 08:26 AM

After getting through "2.L. Sound" I went on a bit of a detour and wrote a really rudimentary ESM / ESP reader / writer that I'd like to use to do plugin cleaning automatically using just a list of specific changes to make defined in the XML file.

 

One annoying part is that apparently instead of just removing the ITMs and calling it a day, xEdit (at least 3.1.3) also appears to change the order of items in the neighborhood of what it touches.  Completely untouched top GRUPs, however, seem to be left alone.  So there are a few options for how this tool can handle deleting ITMs:

  1. Only delete ITMs, and just let the "cleaned" file have a different MD5 than if it were to go through xEdit's cleaning process
  2. Add commands to reorder things within the file
  3. Have this tool's "delete these ITMs" automatically reorder things the way xEdit would have done

I don't think #3 is reasonable without adding deeper semantic understanding (which is expensive for me to write, and makes it slower to process), and #2 gets tricky because some GRUPs (mainly "world children" sub-GRUPs and their own sub-GRUPs) can only be identified by tagging lots more reader context.  #1 gives me pause, because I'm not sure how I feel about pressing forward with adding yet another MD5 checksum that denotes a perfectly valid plugin file unless this gets popular.

 

xEdit also seems to make other unusual changes to the plugin file when it's asked to remove ITMs.  For example, in Update.esm, the field data for [WEAP:000BE25E] seems to get changed for no apparent reason.  Maybe this is a bug in xEdit?  I don't know.

 

So that's where I am right now.  For the record, after sorting the things that xEdit sorts and tweaking the questionable tweaks that xEdit does, I'm able to produce a file with an identical MD5 checksum to what xEdit produces when it removes ITMs, so it's definitely worth considering going further down this path instead of going straight to something like AutoHotkey for it.


  • 0

#13 Greg

Greg

    High King

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,420 posts

Posted 27 September 2016 - 06:02 PM

If I'm reading this correctly, WEAP/000BE25E/DNAM/Flags1 changes from DontUseFirstPersonISAnimations to None and WEAP/000BE25E/DNAM/Flags3 changes from DontUseThirdPersonISAnimations to None. I have no idea why these two flags change when Update.esm is cleaned, though.



#14 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 108 posts

Posted 30 September 2016 - 08:38 AM

I think I'm going to set some goals for the automated cleaning, because this just gets weirder and weirder as I look more and more into what (at least v3.1.3 of) xEdit does when it cleans Update.esm.

 

Goals are:

  1. Remove the same records and groups that xEdit removes when we ask it to remove ITMs
  2. Make the same changes to REFR records that xEdit makes when we ask it to do UDRs
  3. Catalog all the other changes that xEdit makes and figure out which ones this tool should also make and which ones it can skip

I think I can actually automate #1 and #2 entirely, without much hardship on my part, by modeling a complete version of the base (layering specified ESMs and ESPs one on top of the other, in a specified order) and using some slightly "smart" logic to figure out how to write a "cleaned" version of the "dirty patch" into the ModOrganizer folder by only specifying something like "Clean Update.esm using Skyrim.esm as a base", "Clean HearthFires.esm using Skyrim.esm as a base", "Clean Dawnguard.esm using Skyrim.esm + Update.esm as a base", where to find those input files, and where to place the output files.

 

#3 will probably wind up being a post in a different forum.  My intuition is that most or all of the questionable changes don't actually need to happen (maybe they're incorrect, maybe they're superfluous), but there's a chance that a few of these are doing something useful.  It seems like any useful / important changes shouldn't be tacked-on at the plugin cleaning step, but rather they should be incorporated into USLEEP or something.


  • 0

#15 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,794 posts

Posted 30 September 2016 - 12:35 PM

Just post in the xEdit topic or PM Zilav about the odd changes. He's on here from time to time and will see it. If they are bugs, then having him correct those bugs will benefit all users. Else, he'll be able to tell you why the changes are made.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users