Jump to content


Photo

Automated STEP Installer - Now in Beta


  • Please log in to reply
191 replies to this topic

#1 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 13 October 2013 - 12:08 PM

The Automated STEP Installer is now in beta testing. The program installs archives into a dummy skyrim directory, and then archives the contents of that skyrim directory into a 7z file. This 7z file can be installed in NMM or MO. For MO, there will be a popup saying the mod structure is not correct and it will bring up a file configuration menu. Uncheck skyrim, left click on data, and select "set data directory". Make sure all the new boxes are check and then click "ok". This will properly install the created archive. 

The installation folder for ASI is as follows:

A directory contains all mod archvies. This might be called "STEP [version name]". 

In this directory all STEP mod archives will be placed, AutomatedSTEPInstaller.exe will be placed, and 7z.exe and 7z.dll will be placed. 

A subfoler in that directory called "source" must exist. In that directory there is three files. These files have no extensions (so no .txt). The name of the files and what they means is as follows:

format of config file


Format of name file
Whatever you put in this file is what the archive will be called at the end. A good name might be "STEP [version number]". At the end a 7z archive containing all the mod files in the proper data structure will be created with that name. 

Format of sourceDirectory
This file tells the program where to extract everything too. Unless you have a good reason, it should contain the name "skyrim" and nothing else. 

Basically here is an overview of how the program works. It reads in the name of an archive. If the archive is a normal type, meaning it doesn't begin with a "!" character, it extracts it into the skyrim directory. It then looks for directories that aren't called "data" or "textures" or "meshes" or anything you would expect in the skyrim folder, and it moves all the files and folders out of these folders into the skyrim folder, then deletes that folder. So if an archive had this structure

archivename/data/meshes/morestuff




The skyrim directory would have the following structure
meshes
data




The program than moves all folders that aren't name "data" into the data folder. So after that happens, the above skyrim folder would look like this:

data




It also moves all .esp, .bsa and .bsl files into the data folder as well, provided that where only one level down in the original folder they came from, or where extracted into the skyrim folder to begin with. 

When an archive doesn't have a nice structure, and it needs to have special stuff done to it, it will begin with a "!" character. This archive will not be extracted to the "skryim" folder, but to a "temp" folder instead. All lines following the name of that archive will be windows command line instructions, a sort of mini script, that installs that mod. This has to be written by someone at step. This program will resume normal operations when it encounters a line that reads "!". For a more detailed description of how this works, see the config format section of the OP

When it has gone through all the archives in the config file, it will zip everything up in the skyrim folder into an archive. This archive can be installed by NMM or MO or what have you. For MO, it will tell you the archive does not have the correct structure and bring up a file menu. Uncheck skyrim, left click on data, and select "set data directory". Make sure all the new boxes are check and then click "ok". This will properly install the created archive.

If you have any questions about the program or how it works please post them. I will continue to add to this OP to make everything as clear and concise as I can. It will take some getting used to, but once you do this program will prove easy to use and very powerful. 

The latest version of ASI can be found here:

http://www.mediafire...32b9o9ivop1tg5a
  • 0

#2 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 13 October 2013 - 12:34 PM

Imagine a script that unpacks all the rar files, takes out only the files that it needs, and builds a skryim folder that can be archived and installed by a mod manager like MO or NMM, or simply copied into the skyrim folder directly. The user of STEP would just have to put all the rar files in one place so the script can know where they are. If we wanted to update STEP, we would just need to updated our huge file list. There wouldn't be questions anymore about which mod files are supposed to to overwrite which other mod files, because the master list of files would take care of all that. Ok, now the questions is, how do we go about creating a master list of files? How should the script be programmed to process the list?


Have a look at my last post on the other thread on this subject.  This is already possible with my 'Skyrep" scripts in the Semi Auto forum.

The problem isn't the mechanism for doing it - it's the interest of people to maintain it.
  • 0

#3 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 13 October 2013 - 12:41 PM

Can you link me to the post? I will volunteer to maintain it if need be. If your script works similar to the script I had in mind, it won't be all the difficult to do.
  • 0

#4 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 13 October 2013 - 12:45 PM

Here.

And here.
  • 0

#5 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 13 October 2013 - 01:11 PM

I stated this on the other thread, but Monty is essentially correct: Maintenance is the barrier. An automated installer is not possible, given the prerequisites; therefore, a semi-automated procedure was basically devised. This requires one or more people to maintain the installer for each and every input change (and they occur almost daily, sometimes in multiples) in order to maintain the "automation" for the end user. I'll be blunt: that kind of maintenance for "no frills" end-user support is a for-profit enterprise. I fundamentally disagree with hand-holding for nothing. Why should a few of us maintain such a convenience at such great cost to other activities that are more rewarding? What about mod testing as tech aptly mentions? What about wiki and forum maintenance and keeping this community alive and well? We can catch the fish and feed the people, or we can simply teach them how to fish. The latter is much more sustainable, and it is much bigger and much more "alive". Users should be required to help out ... this is a community project ;)

#6 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 13 October 2013 - 01:24 PM

I mostly agree with that. There are schemes and schedules we could use to ease and regulate the update and maintenance burden, but the evidence of history suggests that the work would still fall to a very few - witness the uptake for the request for alpha testing!
  • 0

#7 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 13 October 2013 - 05:20 PM

[quote name=''MontyMM' pid='53414' dateline='1381688661']I mostly agree with that. There are schemes and schedules we could use to ease and regulate the update and maintenance burden' date=' but the evidence of history suggests that the work would still fall to a very few - witness the uptake for the request for alpha testing![/quote']
... alpha testing Packs has similar lack of uptake ;)

The percentage of the populace that wants to actually mod and contribute to modding solutions is rather small. Most of the community wants to just hurry up and install a bunch of mods according to some golden recommendation (one that includes as many mods and changes as possible no less). They do want that "plug & play" guide with download and installation solutions, but they can't have it unless they are willing to get their hands dirty and commit to process as well as product by getting involved in the mechanics and maintenance of it all.

Neovalen's project will be instructive in this regard. It will be relevant for a week or two after its final release (next year maybe?), but without constant maintenance, it will quickly fall into irrelevance ... who knows, maybe there will be those willing to do that work, but I am not going to hold my breath ;)

The STEP:Core + Packs paradigm is one way to allow community contribution with relatively low maintenance from our core team, but I don't see any clear path to a more stable solution until Skyrim modding is no longer an attractive passtime :P

In the meantime, I think that productive, self-starting persons should develop and maintain their projects as they see fit and mock up and deploy a working solution if they have one (that is what the wiki and forums are for). We can all talk ourselves blue with great ideas, but it always comes down to implementation (STEP Core + Packs is basically implemented as a useable alpha right now, just need people to begin using it and providing feedback so that we can maximize it and perhaps achieve a viable end result)

#8 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 13 October 2013 - 07:16 PM

Don't worry - I haven't forgotten about the packs!  ;)  I'm still playing around with the basic setup of my pack (looking at some alternative options, like some modules of ERSO instead of SU), but I'll upload and mess with the wiki as soon as I've settled on a rough draft. If there's evidence of a surge of interest in the tools I was putting together, I'll work with whoever's willing to put time in.  Readers may have noted that I'm not typically backwards about arguing my corner :whistling: , but I saw this one as something of a "put up or shut up" situation - was I willing and able to put in all the extra time to make the system work by myself?  Nope, and it would have been a bit of Don Quixote performance since no-one else appeared to be interested! To mothergoose and phazer, and whoever else might be keen, there are a couple of possible remedies to chasing daily updates with a system that has to be version-specific.  The first and simplest would be to contact the mod authors, tell them that they're included in STEP, and explain why it would be helpful to leave old versions of their mods in the Nexus section provided.  If they co-operated, updates to automatic build could be much more relaxed, and it's not really a huge ask. Another option would be to have a schedule of 'jumping-on points' for automated STEP versions.  What I mean by that, is that we could say on the Friday of Week 'A' , everyone downloads the mod versions available at that time, and the next update of the automatic build will then be 'frozen' to use only those versions, giving everyone a chance to acquire them at that point.  Then, in Week B, having given a chance for the maintainers to update their build based on those versions, the updated patch is released. That way, any newbies wanting to join the scheme would simply wait for the next jumping-on day to get in on the action. I actually think this sort of release routine (which could be independent of the STEP guide releases, focussed on version changes, not content changes) could be quite helpful.  If using the scripts I put together, this would actually require little effort, since it would simply be a case of replacing the agreed updated mods in a master build of a STEP data folder (probably using Wrye Bash to keep it nice and clean) and then clicking to generate a new patch once the updated mods are in place.  If we had a team of maintainers, we could even share the work of maintaining a master build of a STEP Skyrim installation, synchronised between the team using something like Rsync or Crashplan, and a shared changelog when we contribute a change. However, make no mistake - this is extra work and commitment for whoever takes it on, and I can't sign up for that myself at the moment. So beware! :psyduck: :devil: :psyduck: < (that's me, and whoever ends up doing the maintenance. :P
  • 0

#9 phazer11

phazer11

    Chatroom Supervisor

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 3,274 posts

Posted 14 October 2013 - 09:01 PM

Hmm... sounds interesting. I have a bit on my plate atm projects and such I probably won't be free till a little before or after thanksgiving, then I'll have about two months of nothing but free time. I wonder how difficult it would be to take your work Monty and build a ModOrganizer plugin that puts the files it downloads through say the NMM downloader functionality (downloading the whole files possibly then extracting and discarding the excess files something like a batch file that downloads the files through torrent and then extracts and repackages, like a franken-torrent-(.)bat file). Also curse you for using Psyducks I'm hanging by a thread trying to keep myself from digging out the VHSs and watching Pokemon Season 1.

#10 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 15 October 2013 - 03:32 AM

I haven't looked at MO in this regard - I don't know how its plugin system works, and what functionality is easily exposed. One of the things that I was thinking about, when I imagined this would be a popular scheme, was that its best form might be as a sort of "mod-collection organiser". The distinction between that and the mod managers we have today, is that they are focussed on creating the final playable subset of mods you've downloaded. A mod collection organiser, by contrast, would be much simpler, more focussed on managing your whole library of downloaded mods, and manipulating the collection. It would be a sort of permanent library, that could handle version histories and so on. The automation scheme would then operate as one feature of this tool. I would see the automation being implemented by definition files that would be downloaded separately, and executed by this tool (I had a rough schema worked out for these definition files, essentially a Skyrep patch, a list of required mods, and some ini files, wrapped up in an archive.) A definition file might be for, say, Monty's Mad Mods 2.0, or any other complete integrated collection you could dream up. The library would be examined, and you would be informed of any mod archives missing from your library that were needed for that definition. Once the requirements were satisfied, the tool would use the definition to create copy of those mods, transformed into a perfect implementation - restructured, renamed, patched, etc, ready to be used. (It would also be possible to define BSA extraction, and possibly texture optimisation too.) Updates to the definitions would recognise existing transformed files, and not duplicate the work. Part of my thinking here, was not about providing a product just for the end 'consumer'. I do think that even for the willing enthusiast, including parts of the team (certainly this part), it becomes a bit of a drag maintaining one's collection and monitoring updates, etc. By collaborating on automated definition files, we essentially split that effort and keep ourselves in sync. I would envisage having a simple table on the wiki, where the community could help monitor and record updates to the mods we use - making it much easier to define and create updated definitions. But, the perennial problem we face is that, over time, it is only ever a small handful that do the maintenance. And so, it ceases to be a helpful community tool, and becomes more what Z described - a burden on a few poor schmucks toiling to maintain a free service.
  • 0

#11 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 October 2013 - 10:24 AM

@Monty This is a useful tool that you just described. I and others would definitely find it convenient for maintaining and versioning a well defined mod list like STEP:Core, STEP:Extended and any Packs that become established. First trial run would be on STEP:Core of course (since that is the only very well defined mod list right now), and for this, I would be an active participant, as would tech I imagine ... it would be great to get similar commitment from the rest of the team if we can get it polished well enough for daily use.

#12 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,797 posts

Posted 15 October 2013 - 11:32 AM

No comment thus far besides it just sounds like even more maintenance for a small group of us keeping STEP alive.

#13 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,245 posts

Posted 15 October 2013 - 01:30 PM

@Monty

This is a useful tool that you just described. I and others would definitely find it convenient for maintaining and versioning a well defined mod list like STEP:Core, STEP:Extended and any Packs that become established.

First trial run would be on STEP:Core of course (since that is the only very well defined mod list right now), and for this, I would be an active participant, as would tech I imagine ... it would be great to get similar commitment from the rest of the team if we can get it polished well enough for daily use.

No comment thus far besides it just sounds like even more maintenance for a small group of us keeping STEP alive.

but see above for a method to keep STEP:Core synced for those of us that are actively maintaining it. I am thinking that this could actually be a tool to make that task simpler.

#14 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,797 posts

Posted 15 October 2013 - 01:43 PM

You all know that programming isn't my thing so I'm not going to pretend I understand what will go on under the hood of such a tool. At this point there have been several ideas tossed around in this thread. Is there an outline detailing this tool, what it will do and its benefits? Something that I can understand.

#15 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 15 October 2013 - 02:56 PM

First of all, I think it's very important not to have pie-in-the-sky ambitions for something like this. The tricky part is in the Skyrep patches that do the transformation, and you don't need to worry too much about how it does its voodoo. Essentially, it was originally a linux app by a well-known guy in the unix world, called Lennart Poettering. He designed it to synchronise media libraries across PCs using patches that would describe folder structures and binary changes, for maximum efficiency when transmitting the change data from one PC to another. I just ported it to Windows, and modified it a bit so it would operate without prompting for user interaction. Then I added a some batch scripts to add some logic to tailor it for mod archives. That, by far, is the tricky part taken care of. But, the point at which I shelved it was still fairly embryonic - not an end-user friendly product, but just the basic mechanisms that needed testing to make sure others could replicate my results. The next part to build would be the collection manager I mentioned. This would essentially just be a GUI for a straightforward database containing your downloaded mods, and some metadata. This would be nowhere near the complexity of powerful mod managers like MO and Wrye - it would just be to keep your downloaded mod archives in an organised state. Then, the automated patches we could build with Skyrep (which would compare a complete STEP data folder to the collection of mods used to create it) would be executed as a feature of this GUI database app. It would look at the end-user's collection of downloaded mod archives in his database, tell him if he was missing specific ones required by that version of the patch, and then create edited copies of those mod archives to perfectly match the model data folder of the patch creator. The immediate benefit to the team, is that we could all contribute to the maintenance of a master STEP build, kept current with updates. This would mean that even if there were only two people equally sharing the maintenance, they would still be doing half as much work as they would otherwise be to keep their own STEP builds up-to-date. Then, by distributing the change patches to users, you make the project far more accessible to newcomers, and could also use it as a reliable and consistent baseline for support. So, it actually has the potential to reduce workload more than add to it. But that is only true if the maintainers are working to keep their own STEP builds up to date anyway - if they are not, then of course it must create additional work compared to the status quo.
  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users