Jump to content
  • 0

Python Scripts (Wrye Bash)


CptJesus

Question

Included in this post are some fun little python scripts that I've been using for the STEP install. All of the scripts deal with repacking archives to 7zip maximum compression with solid archives turned off. These are tuned towards Wrye Bash, because that's my tool of choice for this process. All of these require the 7zip binary in the same folder as the scripts, as well as Python installed. I've tested all of these with Python 3.x, but it's entirely possible they may work with 2.x. The scripts have a variable called FOLDERPATH, which should point to your Skyrim Mods folder. 

 

archivefixer.py

import sys
import subprocess
import os
import shutil
import re

FOLDERPATH = "X:\Steam\steamapps\common\Skyrim Mods\Bash Installers"

def main(argv):
   if (os.path.exists("temp")):
      shutil.rmtree("temp")
   infile = argv[1]
   outfile = os.path.splitext(infile)[0]
   startdir = os.getcwd() + "\\"
   zippath = os.path.abspath("7z.exe")
   outputpath = startdir + outfile+".7z"
   infile = startdir  +  infile
   origsize = os.stat(infile).st_size
   cmd = [zippath,'x','-otemp',infile]
   subprocess.call(cmd)
   os.remove(infile)
   traverse("temp",zippath,outputpath)
   os.chdir(startdir)
   shutil.rmtree("temp")
   finalsize = os.stat(outputpath).st_size
   shutil.move(outputpath,FOLDERPATH)
   saving = (origsize - finalsize) / 1024
   print("Total Bytes Saved: " + str(saving) + "kb")

def traverse(path,zippath,outputpath):
   os.chdir(path)
   filenames = os.listdir(os.getcwd())
   for filen in filenames:
      if re.match(".*esp",filen) or re.match(".*esm",filen):
         print("We've found the directory with the esp! Zip it all up")
         cmd = [zippath,'a','-ms=off','-mx9',outputpath,'*']
         subprocess.call(cmd)
         return
      elif re.match("00.*",filen) or re.match("01.*",filen):
         print("Found a directory with Wrye Bash Stuff. Zip it all up!")
         cmd = [zippath,'a','-ms=off','-mx9',outputpath,'*']
         subprocess.call(cmd)
         return

   filenames = [x.lower() for x in filenames]

   if ("meshes" in filenames or "textures" in filenames or "interface" in filenames or "video" in filenames or "strings" in filenames or "sound" in filenames or "scripts" in filenames):
      print("We've found the meshes/textures directory! Zip it all up")
      cmd = [zippath,'a','-ms=off','-mx9',outputpath,'*']
      subprocess.call(cmd)
      return
   elif ("skse" in filenames or "SKSE" in filenames or "Skse" in filenames):
      print("SKSE Plugin. Zip it!")
      cmd = [zippath,'a','-ms=off','-mx9',outputpath,'*']
      subprocess.call(cmd)
      return
   elif (len(filenames) > 0):
      for filen in filenames:
         if (os.path.isdir(filen)):
            traverse(filen,zippath,outputpath)
            os.chdir("..")
      return
   else:
      return

if __name__=="__main__":
   main(sys.argv)

 

 

 

 

This script will take any archive, unpack it and then recompress it. This is meant for simple archives that don't need repacking. It will find Wrye Bash folders, as well as simple archives with textures/meshes/sounds etc. Also ESP file. It takes a single archive and will repack it with the same name that it came in with. It'll move the final archive to FOLDERPATH. For example:

 

archivefixer.py "HQ Shields-225-1.7z"

 

magicpacker.py

import sys
import os
import shutil
import glob
import subprocess
import re

FOLDERPATH = "X:\Steam\steamapps\common\Skyrim Mods\Bash Installers"

DUALSHEATHLIST = {"Dual Sheath Redux-34155-1-6b.7z":"00 Core","Elemental Staffs Pack-34155-.7z":"01 Elemental Staffs Patch","Skyrim Weapon De-LARP-ification Project Pack On Back Update-34155-.7z":"02 Skyrim De-LARP Patch"}
NACMIM = {"NACMIM - Full-26822-1-6.zip":"00 Core","NACMIM SkyUI 3-4 Patch - Default-26822-1-34.zip":"01 SkyUI Patch"}
SKYHD = {"Skyrim HD v1_5 LITE - Dungeons-607.7z":"00 Dungeons","Skyrim HD v1_5 LITE - Towns-607.7z":"00 Towns","Skyrim HD v1_5 LITE - Landscape-607.7z":"00 Landscape","Skyrim HD v1_5 LITE - Misc-607.7z":"00 Misc"}
SERIOUSHD = {"Serious HD Retexture LANDSCAPE 1024px-2146-v2-0.rar":"00 Core","Serious HD Retexture RIFTEN 1024px-2146.rar":"01 Riften Patch"}
TOBE = {"Tobes Highres Textures 1_2 1024-1123-1-2.rar":"00 Core","Tobes Highres Textures 1_2b SMIM Patch-1123-1-2.rar":"01 SMIM Patch"}
FOOTPRINTS = {"Footprints v0_99-22745-0-99.7z":"00 Core","Footprints v0_99 - Ash Supplemental-22745-0-99.7z":"01 Ash Supplemental"}
LANTERNS = {"Lanterns Of Skyrim - All In One - MCM Special Edition-18916-2-3.rar":"00 Core","Lanterns Of Skyrim - All In One - Default preset-18916-.rar":"01 Preset"}
MUSHROOM = {"Mushroom retexture pack-29935-1-1.rar":"00 Core","mushroom stew-29935-1.rar":"01 Mushroom Stew"}
VILLAGEANIMALS = {"More Village Animals-8565-2-1.rar":"00 Core","More Village Animals Ivarstead Extended-8565-2-1.rar":"01 Ivarstead",
"More Village Animals Rorikstead Extended-8565-2-1.rar":"02 Rorikstead"}
BETTERMALEFEET = {"Better Male Feet-32378-.rar":"00 Core","Dawnguard Patch-32378-.rar":"01 Dawnguard"}
CROSSBOW = {"Explosive Bolts Visualized v1_0_2-21922-1-0-2.7z":"00 Core","Explosive Bolts Visualized HD Textures-21922-1.7z":"01 HD Patch"}
ELEMENTALSTAFF = {"Elemental Staffs 1K-15152-1-00.rar":"00 Core","Elemental Staffs Update-15152-1-10.rar":"01 1.10 Patch"}
DELARP = {"Skyrim Weapon DeLARPification Project v1_0_5-16072-1-0-5.7z":"00 Core",
"Skyrim Weapon DeLARPification Project Dawnguard v1_0-16072-DG1-0.7z":"01 Dawnguard","Skyrim Weapon DeLARPification Project Dawnguard v1_0_1-16072-1-0-1.7z":"02 Dragonborn"}
GREATSWORD = {"Greatswords SSwTB v2_3-27178-2-3.rar":"00 Core","Greatswords SSwTB - Dawnguard v2-27178-2.rar":"01 Dawnguard",
"Greatswords SSwTB - Dragonborn-27178-1.rar":"02 Dragonborn"}
CLOSEHELMET = {"Improved Closefaced Helmets --FULL SET-- V0_9-15927-0-9.zip":"00 Core",
"Improved Dawnguard helmets-15927-0-9b.zip":"01 Dawnguard","improved imperial full face helmet - no cloth around the neck--15927-0-9.zip":"02 No Cloth"}

def processarchive(archivename,outputdir):
   print("Extracting Archive " + archivename + " to " + outputdir)
   tempcwd = os.getcwd()
   os.makedirs(outputdir)
   cmd = ["7z.exe",'x','-otemp',archivename]
   with open(os.devnull, "w") as q:
      subprocess.call(cmd,stdout = q)
   os.remove(archivename)
   out = os.path.abspath(outputdir)
   traverse("temp",out)
   os.chdir(tempcwd)
   shutil.rmtree("temp")

def traverse(path,outputpath):
   os.chdir(path)
   filenameso = os.listdir(os.getcwd())
   founddir = False
   for filen in filenameso:
      if re.match(".*esp",filen) or re.match(".*esm",filen):
         founddir = True

   filenames = [x.lower() for x in filenameso]

   if (founddir or "meshes" in filenames or "textures" in filenames or "interface" in filenames
      or "video" in filenames or "strings" in filenames
      or "sound" in filenames or "scripts" in filenames):
      for filex in filenameso:
         shutil.move(filex,outputpath)
      return
   elif (len(filenames) > 0):
      for filen in filenames:
         if (os.path.isdir(filen)):
            traverse(filen,outputpath)
            os.chdir("..")
      return
   else:
      return

def testlist(filenames,test,listname):
   testset = set(test.keys())

   if (testset.issubset(filenames)):
      print("Found files for " + listname)
      templ = []
      for k in testset:
         processarchive(k,test[k])
         templ.append(test[k])
      print("Compressing archives to our new package: " + listname + ".7z")
      cmd = ["7z.exe",'a','-ms=off','-mx9',listname + ".7z"]
      cmd.extend(templ)
      with open(os.devnull, "w") as q:
         subprocess.call(cmd,stdout = q)
      for t in templ:
         shutil.rmtree(t)
      print("Moving our file to its new home!")
      shutil.move(listname + ".7z",FOLDERPATH)

def main():
   filenames = glob.glob('*.rar')
   filenames.extend(glob.glob('*.7z'))
   filenames.extend(glob.glob('*.zip'))

   converted = set(filenames)

   testlist(filenames,DUALSHEATHLIST,"Dual Sheath Redux")
   testlist(filenames,NACMIM,"Not Another Colored Map Icon Mod")
   testlist(filenames,SKYHD,"Skyrim HD")
   testlist(filenames,SERIOUSHD,"Serious HD")
   testlist(filenames,TOBE,"Tobe's Highres Textures")
   testlist(filenames,FOOTPRINTS,"Footprints")
   testlist(filenames,LANTERNS,"Lanterns of Skyrim")
   testlist(filenames,MUSHROOM,"Realistic Mushrooms")
   testlist(filenames,VILLAGEANIMALS,"More Village Animals")
   testlist(filenames,BETTERMALEFEET,"Better Male Feet")
   testlist(filenames,CROSSBOW,"Explosive Bolts Visualized")
   testlist(filenames,ELEMENTALSTAFF,"Elemental Staffs")
   testlist(filenames,DELARP,"Skyrim Weapon De-LARPification Project")
   testlist(filenames,GREATSWORD,"Greatswords Sheaths and Scabbards")
   testlist(filenames,CLOSEHELMET,"Improved Closefaced Helmets")

if __name__=="__main__":
   main()

 

 

 

This is designed to help re-compress archives that have patches with them. It's already preconfigured with a lot of the complex archives from STEP. Simply download the main archive and the patches, and then run magicpacker.py and it'll automagically make a nice Wrye Bash archive. I'm in the process of adding all the patched archives, but it's going to take some time to go through the whole thing.

 

wizardparser.py

import sys

def main(argv):
f = open(argv[1],"r")
o = open("wizard.txt","w")

for line in f:
   newline = line.replace("REQUIRE","RequireVersions \"\", \"\", \"\", \"296+\" ; Skyrim version, SKSE version, (no SKGE), Wrye Bash version")
   newline = newline.replace("SM","SelectMany")
   newline = newline.replace("SO","SelectOne")
   newline = newline.replace("SSP","SelectSubPackage")
   newline = newline.replace("DSA","DeSelectAll")
   newline = newline.replace("BRK","Break")
   newline = newline.replace("ES","EndSelect")
   newline = newline.replace("IMG","Wizard Images\\\\")
   newline = newline.replace("DEF","Default Texture")
   newline = newline.replace("CUST", "Custom Texture")
   newline = newline.replace("NOIN","Do not install a texture")
   newline = newline.replace("\"CTEX","\"Customized Texture from")

   o.write(newline)

if __name__=="__main__":
   main(sys.argv)

 

This script is designed to help simplify the process of making Wrye Bash Wizards by converting shorthand to a full Wizard. All it takes is an input file, and it will output wizard.txt. Here's an example of input:

 

REQUIRE

DSA

SO "Select Configuration for Book of Silence (Weapons)", \
   "|No Flesh (STEP Recommended)", "The recommended install for Step", "IMGOpt 4 - No Flesh.jpg", \
   "Option 0", "","IMGOpt 0.jpg", \
   "Option 1", "","IMGOpt 1.jpg", \
   "Option 2","","IMGOpt 2.jpg", \
   "Option 3","","IMGOpt 3.jpg"

   Case "No Flesh (STEP Recommended)"
      SSP "00 Option 4 - No Flesh"
   BRK

   Case "Option 0"
      SSP "00 Option 0"
   BRK

   Case "Option 1"
      SSP "00 Option 1"
   BRK

   Case "Option 2"
      SSP "00 Option 2"
   BRK

   Case "Option 3"
      SSP "00 Option 3"
   BRK
ES

This will convert to:

 

RequireVersions "", "", "", "296+" ; Skyrim version, SKSE version, (no SKGE), Wrye Bash version

DeSelectAll

SelectOne "Select Configuration for Book of Silence (Weapons)", \
   "|No Flesh (STEP Recommended)", "The recommended install for Step", "Wizard Images\\Opt 4 - No Flesh.jpg", \
   "Option 0", "","Wizard Images\\Opt 0.jpg", \
   "Option 1", "","Wizard Images\\Opt 1.jpg", \
   "Option 2","","Wizard Images\\Opt 2.jpg", \
   "Option 3","","Wizard Images\\Opt 3.jpg"

   Case "No Flesh (STEP Recommended)"
      SelectSubPackage "00 Option 4 - No Flesh"
   Break

   Case "Option 0"
      SelectSubPackage "00 Option 0"
   Break

   Case "Option 1"
      SelectSubPackage "00 Option 1"
   Break

   Case "Option 2"
      SelectSubPackage "00 Option 2"
   Break

   Case "Option 3"
      SelectSubPackage "00 Option 3"
   Break
EndSelect

 

Example usage is wizardparser.py test.txt

 

The following shorthand is used:

 

REQUIRE is the Require String

SM = SelectMany

SO = SelectOne

SSP = SelectSubPackage

DSA = DeSelectAll

BRK = Break

ES = EndSelect

IMG = Wizard Images\\

DEF = Default Texture

CUST = Custom Texture

NOIN = Do not install a texture

CTEX = Customized Texture from

 

 

Hope some of these scripts help out you STEPpers

Link to comment
Share on other sites

Recommended Posts

  • 0

So I'm sure you guys are getting really tired of seeing these scripts, but I threw together another script that I thought would be useful. It's basically complete functionality wise, but I'm working on a dictionary to go with it. I'm calling it the MagicPacker. I noticed that a lot of STEP mods have multiple items, such as patches. When this is complete, you should theoretically be able to download all your mods and just run this, and it'll repackage any complex archives into nice Wrye Bash ones. It uses a dictionary to create mappings for Archives to Folders, and then attempts to group them together. If it doesn't find all the archives necessary for STEP, it'll just skip it. As of right now, the only one configured is Dual Sheath Redux.

 

The issue with this packer is going to be maintenance as file names update. Let me know what you guys think. I'm going back through the STEP install again, and I'm actually going to set multiple cases up and see how it works out.

import sys
import os
import shutil
import glob
import subprocess
import re

FOLDERPATH = "X:\Steam\steamapps\common\Skyrim Mods\Bash Installers"

DUALSHEATHLIST = {"Dual Sheath Redux-34155-1-6b.7z":"00 Core","Elemental Staffs Pack-34155-.7z":"01 Elemental Staffs Patch","Skyrim Weapon De-LARP-ification Project Pack On Back Update-34155-.7z":"02 Skyrim De-LARP Patch"}

def processarchive(archivename,outputdir):
   print("Processing Archive " + archivename + " to " + outputdir)
   tempcwd = os.getcwd()
   os.makedirs(outputdir)
   cmd = ["7z.exe",'x','-otemp',archivename]
   subprocess.call(cmd)
   os.remove(archivename)
   out = os.path.abspath(outputdir)
   traverse("temp",out)
   os.chdir(tempcwd)
   shutil.rmtree("temp")

def traverse(path,outputpath):
   os.chdir(path)
   filenameso = os.listdir(os.getcwd())
   founddir = False
   for filen in filenameso:
      if re.match(".*esp",filen) or re.match(".*esm",filen):
      founddir = True

   filenames = [x.lower() for x in filenameso]

   if (founddir or "meshes" in filenames or "textures" in filenames or "interface" in filenames
      or "video" in filenames or "strings" in filenames
      or "sound" in filenames or "scripts" in filenames):
      for filex in filenameso:
         shutil.move(filex,outputpath)
     return
   elif (len(filenames) > 0):
      for filen in filenames:
         if (os.path.isdir(filen)):
            traverse(filen,outputpath)
            os.chdir("..")
      return
   else:
      return

def main():
   filenames = glob.glob('*.rar')
   filenames.extend(glob.glob('*.7z'))
   filenames.extend(glob.glob('*.zip'))

   converted = set(filenames)

   DUAL = set(DUALSHEATHLIST.keys())

   if (DUAL.issubset(filenames)):
      print("Found files for Dual Sheath Redux!")
      templ =
      for k in DUAL:
         processarchive(k,DUALSHEATHLIST[k])
         templ.append(DUALSHEATHLIST[k])
      cmd = ["7z.exe",'a','-ms=off','-mx9',"Dual Sheath Redux.7z"]
      cmd.extend(templ)
      subprocess.call(cmd)

   for t in templ:
      shutil.rmtree(t)
   shutil.move("Dual Sheath Redux.7z",FOLDERPATH)


if __name__=="__main__":
main()
Link to comment
Share on other sites

  • 0

So before I go to sleep, I'm going to post the update to the previous script. I went ahead and cleaned up the code a bit, and added some more packages (all from baseline STEP). I also supressed the 7zip output because I realized how annoying it was. It takes about 30 seconds to create a new entry to search for and repack, so this is pretty easy to extend. Using this in conjunction with the other repacker script written, you could easily set it up so you just download everything, and then run this and it would repack almost every mod in an intelligent manner and move it over to your Skyrim Mods folder. I'm toying with the idea to speed up reinstalls of STEP and automate a lot of the tedious tasks. Then again, I'm probably one of the few people who actually repack everything to 7zip max and non-solid.

 

import sys
import os
import shutil
import glob
import subprocess
import re

FOLDERPATH = "X:\Steam\steamapps\common\Skyrim Mods\Bash Installers"

DUALSHEATHLIST = {"Dual Sheath Redux-34155-1-6b.7z":"00 Core","Elemental Staffs Pack-34155-.7z":"01 Elemental Staffs Patch","Skyrim Weapon De-LARP-ification Project Pack On Back Update-34155-.7z":"02 Skyrim De-LARP Patch"}
NACMIM = {"NACMIM - Full-26822-1-6.zip":"00 Core","NACMIM SkyUI 3-4 Patch - Default-26822-1-34.zip":"01 SkyUI Patch"}
SKYHD = {"Skyrim HD v1_5 LITE - Dungeons-607.7z":"00 Dungeons","Skyrim HD v1_5 LITE - Towns-607.7z":"00 Towns","Skyrim HD v1_5 LITE - Landscape-607.7z":"00 Landscape","Skyrim HD v1_5 LITE - Misc-607.7z":"00 Misc"}
SERIOUSHD = {"Serious HD Retexture LANDSCAPE 1024px-2146-v2-0.rar":"00 Core","Serious HD Retexture RIFTEN 1024px-2146.rar":"01 Riften Patch"}
TOBE = {"Tobes Highres Textures 1_2 1024-1123-1-2.rar":"00 Core","Tobes Highres Textures 1_2b SMIM Patch-1123-1-2.rar":"01 SMIM Patch"}

def processarchive(archivename,outputdir):
	print("Extracting Archive " + archivename + " to " + outputdir)
	tempcwd = os.getcwd()
	os.makedirs(outputdir)
	cmd = ["7z.exe",'x','-otemp',archivename]
	with open(os.devnull, "w") as q:
		subprocess.call(cmd,stdout = q)
	os.remove(archivename)
	out = os.path.abspath(outputdir)
	traverse("temp",out)
	os.chdir(tempcwd)
	shutil.rmtree("temp")

def traverse(path,outputpath):
	os.chdir(path)
	filenameso = os.listdir(os.getcwd())
	founddir = False
	for filen in filenameso:
		if re.match(".*esp",filen) or re.match(".*esm",filen):
			founddir = True

	filenames = [x.lower() for x in filenameso]

	if (founddir or "meshes" in filenames or "textures" in filenames or "interface" in filenames 
		or "video" in filenames or "strings" in filenames 
		or "sound" in filenames or "scripts" in filenames):
		for filex in filenameso:
			shutil.move(filex,outputpath)
		return
	elif (len(filenames) > 0):
		for filen in filenames:
			if (os.path.isdir(filen)):
				traverse(filen,outputpath)
				os.chdir("..")
		return
	else:
		return

def testlist(filenames,test,listname):
	testset = set(test.keys())

	if (testset.issubset(filenames)):
		print("Found files for " + listname)
		templ = []
		for k in testset:
			processarchive(k,test[k])
			templ.append(test[k])
		print("Compressing archives to our new package: " + listname + ".7z")
		cmd = ["7z.exe",'a','-ms=off','-mx9',listname + ".7z"]
		cmd.extend(templ)
		with open(os.devnull, "w") as q:
			subprocess.call(cmd,stdout = q)
		for t in templ:
			shutil.rmtree(t)
		print("Moving our file to its new home!")
		shutil.move(listname + ".7z",FOLDERPATH)

def main():
	filenames = glob.glob('*.rar')
	filenames.extend(glob.glob('*.7z'))
	filenames.extend(glob.glob('*.zip'))

	converted = set(filenames)

	testlist(filenames,DUALSHEATHLIST,"Dual Sheath Redux")
	testlist(filenames,NACMIM,"Not Another Colored Map Icon Mod")
	testlist(filenames,SKYHD,"Skyrim HD")
	testlist(filenames,SERIOUSHD,"Serious HD")
	testlist(filenames,TOBE,"Tobe's Highres Textures")


if __name__=="__main__":
	main()
Link to comment
Share on other sites

  • 0

I was a little surprised by the requirement to have the 7zip binary in the same folder. I have 7zip in a separate utility directory. Do I have to copy 7z.exe into the directory with the mod files, or can I just add the directory with the 7zip files to the PATH environmental variables?

Link to comment
Share on other sites

  • 0

I've been calling the executable directly, but I'm sure it would work without it with a bit of tweaking. The binary is a remnant from the batch script I was using previously. As it stands right now, you just need 7z.exe in the same folder as the scripts themselves

 

Sent from my SGH-T889 using Tapatalk 2

Link to comment
Share on other sites

  • 0

Just something that may be of interest, that I was looking at a while ago. In the recent (alpha) versions of 7zip, he's added the ability to rename files within the archive from the command line, using the -rn suffix. The folder paths of files in 7zip archives are just considered part of the file name, and renaming the filenames/paths does not require repacking the archive.

 

So, you can get 7zip to dump the filenames/paths of its contents, and then use a a fairly simple script to rename them with the desired logic. Mods only come packed in a limited variety of schemes, so without too much effort you can create a procedure that will restructure almost any mod archive into a Wrye/MO friendly format, using a fairly straightforward set of logical checks on the existing path structures.

 

This, of course, does not help if re-compressing for minimum size is the goal, but it does allow for the restructuring of STEP mods at lightning speeds. In most cases, it should also be possible to add files without decompression/re-compression.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...

Important Information

By using this site, you agree to our Guidelines, Privacy Policy, and Terms of Use.