Jump to content


Photo

Automating S.T.E.P. - Mod Combiner (and automation utility)


  • Please log in to reply
16 replies to this topic

#1 generalmx

generalmx

    Citizen

  • Members
  • Pip
  • 59 posts

Posted 15 January 2013 - 05:49 AM

Hi everyone! I already messaged TheCompiler about automating S.T.E.P., and he said it'd be a good idea to post my results here.

As part of my initial effort to automate S.T.E.P., I've created a configurable replacement to the Texture Pack Combiner's batch script: http://www.skyrim.ne....com/mods/30922

The ZIP file includes the batch file (ModCombiner.bat), and 7zip's command utility (7za.exe) with its license file (license.txt).

This example configuration file is actually the first few lines of the better-known Texture Pack Combiner, converted into the required comma-delimited list. I have the entire batch file converted into one big configuration file that I need permission from the mod author (celestral, I believe) to release.

The batch file works thusly:

- Look for a master ordered list of configuration files to read, one per line. If none found, just go through all configuration files in any order.

- Read through configuration file line-by-line, excluding lines starting with a semi-colon

- Each line of the configuration file contains a file to copy and a destination, separated ONLY by a comma. Example:
Required Mods\HD2K\Textures\actors\dragon priest\dragonpriest.dds,Data\Textures\actors\dragon priest\

Since this is just piping commands to xcopy, we can be a bit more complex in the usage. Example of copying multiple files:
Required Mods\HD2K\Textures\actors\dragon priest\*.*,Data\Textures\actors\dragon priest\

Example of copying a file with renaming:
Required Mods\HD2K\Textures\actors\dragon priest\dragonpriest.dds,Data\Textures\actors\dragon priest\dragonpriest_n.dds

- Lines that start with < are extraction lines containing an archive file and a destination, separated again by a comma.

Example:
<Foo.7z,Foo

Missing archives is not a fatal error and it'll continue processing the configuration file, letting the configuration file have optional archives and different versions.

With reading of configuration files and a master ordered list to support multiple configuration files that must be read in a certain order, I'm hoping this batch file will be highly configurable without needing to ever alter the batch file itself, thus letting people distribute their own configuration files independent of the batch file.

The next step is to add support for downloading files from NexusMods.

Edit: Updated with 7zip support for almost all archive types.

2-5-2013 - Working on both BSA Commander and DDSOpt command-line support, have a working local version that needs testing. I think this'll be a huge help to S.T.E.P. as well as other "patches" people can make using ModCombiner's powerful automation, without needing to potentially violate copyright by re-uploading possibly abandoned works. I'm also looking into what can be done about command-line TES5Edit support and/or diff.
  • 0

#2 ciddative

ciddative

    Prisoner

  • Members
  • 39 posts

Posted 15 January 2013 - 03:09 PM

Sounds terrific! Is the ultimate aim to use this for all conflicting graphics mods in STEP? Cos that's some amount of work right there :P But anything to make our lives easier is a definite plus, keep up the excellent work!
  • 0

#3 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 January 2013 - 03:47 PM

This kind of dev should be merged with the ideas of Thunderbolt so that we can pool ideas and avoid re-creating wheels. @generalmx Are you a developer or scripter?

#4 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,794 posts

Posted 15 January 2013 - 04:41 PM

Sounds terrific!

Is the ultimate aim to use this for all conflicting graphics mods in STEP? Cos that's some amount of work right there :P

But anything to make our lives easier is a definite plus, keep up the excellent work!

I'm already creating a batch file that will work for the Conflicting Graphics section in STEP. I'm half way through, but have paused my work because SMIM has been updated so often as of late. The file will work exactly like it would if you were overwriting the file in a load order, therefore, it completely replaces all the 2.F section mods with one single mod. Later down the line and when the file as been completed, I plan to turn it into more of a TPC-type batch file; picking the best options out of the conflicting textures.

@Generalmx
Thanks for the wild card trick! I don't know why I didn't thank of that. It'll make work go much quicker.

#5 generalmx

generalmx

    Citizen

  • Members
  • Pip
  • 59 posts

Posted 15 January 2013 - 04:56 PM

Sounds terrific!

Is the ultimate aim to use this for all conflicting graphics mods in STEP? Cos that's some amount of work right there :P

But anything to make our lives easier is a definite plus, keep up the excellent work!

Right - the batch file can do that right now, given the correct configuration. It's the configuration files that need to be generated now depending on whatever the latest version of S.T.E.P. is. I have a different batch file not yet distributed which can make configuration files from packages.

This kind of dev should be merged with the ideas of Thunderbolt so that we can pool ideas and avoid re-creating wheels.

@generalmx
Are you a developer or scripter?

Both. I am a big fan of not re-creating wheels, which is one thing I was concerned with Thunderbolt, that they were trying to hardcode a bunch of stuff that doesn't really need to be hardcoded. With S.T.E.P. the way I see it is that you have limited user configuration intended as it's a set of well-researched recommendations, so a GUI isn't really necessary. Now if you want to make a GUI on top of something like this, that's fine and doesn't re-invent the wheel. But the more features you add using all sorts of languages, the harder it is to maintain. This batch file should be quite easy to maintain, with most updates not being to the batch file but to separately distributed configuration files.

Note in one day I've made a highly configurable replacement to the current TPC (where as I'm not sure how far along the current Thunderbolt is - I'm worried it's stuck in "feature creep"); this batch file already handles what I consider at least 50% of what S.T.E.P. needs, the other 50% being downloading, extracting, and installing the mods. Extracting and installing the mods is easy, it's the downloading that will be tough as I'll need to examine NMM's source to figure out how to do that with common tools.

Sounds terrific!

Is the ultimate aim to use this for all conflicting graphics mods in STEP? Cos that's some amount of work right there :P

But anything to make our lives easier is a definite plus, keep up the excellent work!

I'm already creating a batch file that will work for the Conflicting Graphics section in STEP. I'm half way through, but have paused my work because SMIM has been updated so often as of late. The file will work exactly like it would if you were overwriting the file in a load order, therefore, it completely replaces all the 2.F section mods with one single mod. Later down the line and when the file as been completed, I plan to turn it into more of a TPC-type batch file; picking the best options out of the conflicting textures.

@Generalmx
Thanks for the wild card trick! I don't know why I didn't thank of that. It'll make work go much quicker.

On the same note of not reinventing the wheel, you're free to use parts of, redistribute and recredit, and/or change your batch file to a configuration file that can be read by this batch file's logic (comma-delimited lists), noting this can read from any number of provided configuration files. I don't really care much about credit as I have multiple projects going on - these utilities were intended to be released to the community without copyright (public domain).
  • 0

#6 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 January 2013 - 05:22 PM

Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum. Do you code in C++?

#7 generalmx

generalmx

    Citizen

  • Members
  • Pip
  • 59 posts

Posted 15 January 2013 - 05:39 PM

Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum.

Do you code in C++?

Yes, but again I don't think it's necessary for most S.T.E.P. users to have a complicated solution. I'm sure there are people that want a highly configurable solution, but for most people they probably just want something like TPC that at least extracts the mods for them, if not even downloads it for them, as that's the longest process in S.T.E.P. by far.

Most of the maintenance work will be upkeep ordered configuration files, and I can change the logic in the batch file to be the same as Thunderbolt's, as I am a strong supporter of these configuration files to NOT be hardcoded and be very easy for just anyone to edit, so they can make their own. Unless Thunderbolt has some sort of magic image comparison tool that can judge aesthetics, at the end of the day we'll be dealing with an ordered list of files either way.

I am willing to help out with Thunderbolt, but I think I'll keep working on these relatively small and simple batch files as well.
  • 0

#8 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 15 January 2013 - 06:21 PM

Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum.

Do you code in C++?

Fri wasn't planning on having a GUI for Thunderbolt, which I never understood.

#9 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 January 2013 - 06:26 PM

Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum.

Do you code in C++?

Fri wasn't planning on having a GUI for Thunderbolt, which I never understood.

Hmmm, I thought that is what elbe had been setting up as a hub for mass downloading mods from a list, repackaging them in BAIN format and building the custom installers on the fly.

Are you saying you see a GUI for this but Fri did not? Obviously, I like a GUI for this, but don't mind the command line ... but a huge user base won't touch it that way.

#10 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 15 January 2013 - 06:27 PM

Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum.

Do you code in C++?

Yes, but again I don't think it's necessary for most S.T.E.P. users to have a complicated solution. I'm sure there are people that want a highly configurable solution, but for most people they probably just want something like TPC that at least extracts the mods for them, if not even downloads it for them, as that's the longest process in S.T.E.P. by far.

In terms of a GUI, one was planned as a front-end editor. The initial program that was a contender for dealing with things like this was AHC. I still think it was heading in the right direction, it just had some bugs and the editor could use some modification. The general idea was to be able to create rules to not only repackage individual mods, but to even allow combining several mods into a single package. I still believe this is the way to go.

#11 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 January 2013 - 06:32 PM

Yep. Fri hit a kink in the original implementation and decided to rewrite it as Thunderbolt from the ground up using C++. Not really sure what was actually started though. Way back when, MontyMM had developed a mass downloader that could build BAIN packs I think. Then he and elbe hooked up to merge their similar ideas and different approaches .... then ... and then .... here we are.

#12 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 15 January 2013 - 06:34 PM


Thunderbolt is much more than TPC. the GUI is completely necessary, and the plans are all in a hidden forum.

Do you code in C++?

Fri wasn't planning on having a GUI for Thunderbolt, which I never understood.

Hmmm, I thought that is what elbe had been setting up as a hub for mass downloading mods from a list, repackaging them in BAIN format and building the custom installers on the fly.

Are you saying you see a GUI for this but Fri did not? Obviously, I like a GUI for this, but don't mind the command line ... but a huge user base won't touch it that way.

Yeah, elbe had AHC which at least was setup to be able to repackage mods. It didn't have download capability yet though. The last version has a lot of bugs though. But, the concept of AHC is perfect for STEP. Fri didn't like the editor which is why he struck off to create his own, but I've always felt that some polishing of AHC would be all that we need.

As for Thunderbolt, there were plans for an editor at some point for creating custom configurations, so that would require a GUI. His initial plans for the main piece was strictly command line as a simple run-it and wait. I don't like this idea, as it doesn't allow for easy customization. What if a user wants to only install Core mods? Or maybe they want to prevent a few individual mods from installing because they don't like them, can't use them for performance reasons, or want to use something else?

EDIT: Fri was using C#.

#13 generalmx

generalmx

    Citizen

  • Members
  • Pip
  • 59 posts

Posted 21 January 2013 - 07:27 AM

Now the batch file has support for extracting from archives, with an included 7zip utility (and its license file). It will extract archives in the order given to it by the configuration file.
  • 0

#14 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 21 January 2013 - 02:04 PM

We will be ready to tackle this after 2.2.1 release. Let's put it on the back burner right now. There are a great many things to coordinate after 2.2.1.

#15 generalmx

generalmx

    Citizen

  • Members
  • Pip
  • 59 posts

Posted 03 February 2013 - 05:27 AM

Just to note, I went ahead and released what I have now on NexusMods: http://www.skyrim.ne....com/mods/30922 (renaming to "Mod Combiner") ...with the idea being that random people will make their own configuration files for public enjoyment of their personal artistic eye, just like people did with the post-processing effect configurations, etc. (this wasn't intended as only a S.T.E.P. tool) While I think it should be relatively simple for anyone to make a configuration file, I could make one to at least automate the parts of 2.2.1 that require a specific order and/or only specific files from certain mods. You could include the batch file in the download, or just a link. Or I could include it on my mod's download page, with credit to S.T.E.P. I was thinking multiple configuration files, like: Baseline.cfg - Includes baseline/core and files from "Light", main configuration file. Medium.cfg - Optional download addon file for "medium" config. Requires Baseline. Heavy.cfg - Optional download addon file for "heavy" (full) config. Requires Baseline. Insane.cfg - Optional download addon that uses the very largest texture options from mods, probably requiring at least 2GB VRAM. Requires Baseline. With multiple configuration files we give the user extra options as well as ease of updating said files (so you don't need to update Baseline if just a mod in Heavy is effected). My next goal is to add download support and support for the command-line version of ddsopt.
  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users