Jump to content


Photo

getting the metadata


  • Please log in to reply
32 replies to this topic

#1 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 02 December 2013 - 01:12 PM

ok, I think now I've got a rough understanding of how things work "under the hood" in every tool available.

What we need is meta-data for MGs ASI tool to create config files.
I describe the (imho) probably easiest practical way to get those infos. If someone finds a better way doing this I'm always open for suggestions!

Most infos can be easily aquired through tanins Mod Organizer. The tool creates .meta files for each nexus download, which contains those infos in a nice readable ini-like format. So downloading the mods gets 50 % of the job done! But be aware that this function doesn't always work, so there still might be some manual work neccessary.

filenames, fileID, checksums, modID
1st and foremost we need the filenames and checksums of each packed to actually identify them on file-level. A ModID alone won't do it, because many mods have multiple files for download. Then again the modID IS important to connect the fileinfos with general modinfos. Also the MO fileID might be useful for updating the database with more .meta file infos later on.

"Version" info might also help identifying updated archives.

Whats needed is a script that can readout and combine thos infos from .meta files into a data spreadsheet (e.g. csv format)
I'm currently working on this. It will also feature checksum creation for each file. While that's not necessary for a working installation it gives the user feedback if the program actually "knows" on what it's working on and if all files are downloaded correctly.

For ASI we need additional info
a) if its an easy to install file:
files with a correct directory structure are "easy"
-> this could be integrated as a "check" command in ASI. e.g. Check for "data" folder inside .7z etc. the output would be filenames that don't meet the required standards. All "easy" files won't get the advanced copy commands ©. If there isn't a [filename].csv ASI should asume it's "easy" and just extract what it can. Not sure yet if that's good enough or an extra file "easy"  (true, false) is needed in the sheet.

b) internal mod order:
which file will be installed first / overwritten? Think of Mod Version 1 + Mod Update V1.1
-> While we can make a default order by filename there might be mods that won't work that way!


c) the actual copy commands for complicated mods. Not sure about that one yet. probably just a csv with the same prefix as the filename/ or the fileID.
so copy commands for "Enhanced HD Dragon Bones 1k v1_6a all options-28741-1-6a.rar" will be stored in "Enhanced HD Dragon Bones 1k v1_6a all options-28741-1-6a.csv"
CSV data will be stored as
"sourcepath", "destinationpath"
"sourcepath", "destinationpath"
for single files or "y*x" or directories (everything robocopy can handle).


In the end we will have:

1) one database (sheet) with filenames:
filename (text), fileID (int), modID (int), checksum (text), Version (text), modName (text)

edit: added modName for human reader convenience ;)

2) one .csv for each file with advanced copy commands (where needed).


I'm working on 1) and MG is doing 2) (at least as I understood it).

Ready for suggestions and citicism :D

edit1:
Got a small order problem with weapons and armor fixes. This can probably be solved by ordering by fileID first.
  • 0

#2 torminater

torminater

    Dragon Prince

  • Contributors
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2,222 posts

Posted 02 December 2013 - 01:40 PM

That is essentially how I thought we will handle it. Will you release your tool as a MO plugin? Also, any chance you can include a option to create checksums for the files installed in MO's virtual skyrim data folder?
  • 0

#3 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 02 December 2013 - 03:07 PM

That is essentially how I thought we will handle it.
Will you release your tool as a MO plugin?
Also, any chance you can include a option to create checksums for the files installed in MO's virtual skyrim data folder?

good, maybe, maybe ;)

Not sure yet about the necessary adjustments needed for a MO plugin. But this would make a lot of sense.
It shouldn't be a big deal to create checksums of all extracted files once I got it working for the archive files. By "virtual skyrim" you mean the separated "data" directories for each mod? Could be done as well, but imho it would make more sense to combine it with a 7z extract. This way you can be sure the user of the tool hasn't interfered with any file manipulations. As this will be a package manager tool (not for the mods' user) that shouldn't be a problem performance wise. I will include an option to check single .meta/.7z file combos or a whole directory.
  • 0

#4 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,250 posts

Posted 02 December 2013 - 03:25 PM

Just FYI if you don't already know this, but MO has package-structure "check" functionality you refer to ... it seems like any potential redundant features of ASI should borrow those implemented in MO (or you could work with Tannin to improve upon the existing). All sounds logical. Just wanted to point out that there are several interfaces that handle pieces of the bigger puzzle in the OP (SMW and MO for the most part).

#5 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 02 December 2013 - 03:32 PM

The biggest problem I see with using nexus meta data to automate a config file is that the nexus meta data is often wrong. There are so many mod authors that upload an optional file that has nothing to do with STEP, and increment the version number by a tenth. I think keeping organized certainly helps though, and it would be good for that. If there is a wiki page that has the STEP included version number, and the most recent version number, then a SAS guy need only look into a mod if it gets updated past the wiki. As an example, someMod v1.1.1 is in STEP, someMod v1.1.2 is not, when the mod gets updated to v1.1.3 it can be checked against the STEP mod to decide if and how to change it. The mods with special file instructions are a little different. I think its a good idea to keep track of how the config file deals with that. I think an ideal structure would be to describe the file structure of the archive, either in a screenshot, text, or 7z list command export (7z l archive.zip > textfile.txt). This would be along side the exact DOS command for moving the files. This would make it much easier for someone to come in and desk check. Maintainer X might like x copy and know that pretty well, maintainer y might be more familiar with supercopy, and I might come in and do everything with robocopy, if we all know what we are doing the more innocuous details like flags and arguments won't be that important. Another thing to consider, it might make sense to organize the SAS wiki according to the install order, and not the category order like in the regular STEP wiki. The reason being is that the SAS wiki could be made to output a some kind of text file, and then I could write a modified config generator to parse it and build the config file from it. It would make sense to structure the SAS wiki according to the config file that should be built for that reason.
  • 0

#6 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,250 posts

Posted 02 December 2013 - 03:37 PM

SMW does not see mod version at this point (we did not build that in), due to the maintenance hassle of that. We have discussed the potential to include it though if we can make maintenance simple (Nexus API is primary source, regardless of dependability).

#7 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 02 December 2013 - 03:40 PM

You would have to use the nexus API, its the only source for that kind of information. What I am saying is that part of maintaining the wiki would be checking version number against the archives it points too. If we assume the latest version is the archive we want, then there are going to be a ton of problems.
  • 0

#8 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 02 December 2013 - 04:53 PM

yes, the nexus meta data might be wrong in some cases. But this tool is mainly for convinience. It needs to be tested and approved by humans... The idea is to skip the tedious work of writing down every filename + give some extra metadata. This data can then be imported in the wiki for further use and updates. If STEP runs into massive changes, or s.o. wants this for his own package it can be run again. Only requirenment will be the download of all mod files with MO. The idea for a comparism with the "old" STEP mod is great. I maybe even include both: a complete filelist (with checksums) of the zips and a folder structure for easier copy command work. Later a command line switch like "-l" for lite can be included to skip the 7z extraction and just get the usual meta data (+using MGs 7z command for the file structure). edit: recognizing the mod file should be imho done this way (random example file used): - check filename "HQ3DMap - Meshes Hi-Res-4817-2-0.7z" - check CRC - done if filename can't be found, - search for filename* (without version info): "HQ3DMap - Meshes Hi-Res-*" - check folders (to the structure from the db) if folderstructure (or filestructure?) ok: - user message: "The file "HQ3DMap - Meshes Hi-Res-4817-2-0.7z" can't be found. Use "HQ3DMap - Meshes Hi-Res-XXXX" instead?" Answers: yes, no, always yes, always no, search again (/download the old version with MO) But I'm not sure yet if the folder structure would be an appropriate test. Maybe checking if more than 50 % of all files are the same? Anyway, you can't be 100% certain that a new version is compatible - this will always be risky for users. edit2: oh, almost forgot:

  • 0

#9 torminater

torminater

    Dragon Prince

  • Contributors
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2,222 posts

Posted 03 December 2013 - 07:47 AM

Not sure yet about the necessary adjustments needed for a MO plugin. But this would make a lot of sense.
It shouldn't be a big deal to create checksums of all extracted files once I got it working for the archive files. By "virtual skyrim" you mean the separated "data" directories for each mod? Could be done as well, but imho it would make more sense to combine it with a 7z extract. This way you can be sure the user of the tool hasn't interfered with any file manipulations. As this will be a package manager tool (not for the mods' user) that shouldn't be a problem performance wise. I will include an option to check single .meta/.7z file combos or a whole directory.

By virtual Skyrim directory I refer to sth very specific:

All mod packages X are installed by extraction in to MO\mods\X="data folder of mod X". The user generates a profile and enables only certain installed mod packages for this profile. He then has 5 tabs to the right of the MO window:

Plugins = all esps and esms of currently enabled mods plus the vanilla esms with activated and deactivated esps (Load order)

Archives = all BSAs of the enabled mods plus the vanilla bsa archives

-> Data = This is how your Skyrim Data folder looks like, with the current set of enabled mods according to their priority order (which files of which mods win the conflicts between the enabled mods). I asked for a CRC generating plugin with the ability to acquire the info of this virtual Skyrim Data folder (those files never actually get moved to the real Skyrim data folder) and then generating CRC for the "finally installed files".

Saves= your saves of the current profile or all profiles (depends on user config)

Downloads= your downloads downloaded with MO

RE: Data:
If you can provide a plugin that creates those CRC checksums for the files in the virtual Skyrim Data folder, anybody could create a custom Skyrim profile with custom mods and then generate a CRC csv spreadsheet with all the active files and then upload the spreadsheet to the STEP wiki (essentially a "pack"). Now he just needs to give the specific mod versions and anybody using ASI could replicate that exact install (after downloading the required files) in a few seconds.
Depending on the functionality of ASI, it has the capability to tell the user which mod archive (with the exact version) is missing by applying a CRC check on the downloaded archives and comparing them to a second spreadsheet with CRC checksums the pack author uploaded to the wiki as well.


Only downsides for the pack author are:

A) Before the CRC checks are applied:
-No esp cleaning, patching or merging unless they are uploaded to the nexus or wiki.
-No mod renaming during the install (unless that doesn't mess with the checksums and doesn't do any other harm). All archives from a specific Nexus mod page are installed in one MO folder.
-No bsa extraction during installation.

B) All cleaning/merging/patching instructions are then to be included in the pack page and the user follows them manually (or download the required files if they are necessary).

Only downside for the user is:
-No customization options so far (would require additional functionality, like a UI and installer with multiple options). The user only gets those files extracted to MO\mods\X that win the conflicts in the pack author's profile.
  • 0

#10 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 03 December 2013 - 11:36 AM

Only downsides for the pack author are:

A) Before the CRC checks are applied:
-No esp cleaning, patching or merging unless they are uploaded to the nexus or wiki.
-No mod renaming during the install (unless that doesn't mess with the checksums and doesn't do any other harm). All archives from a specific Nexus mod page are installed in one MO folder.
-No bsa extraction during installation


- yes, this will still be a problem. I have some very vague ideas but I'm far from a esp / TES5 expert. From IT pov: you could compare unclean vs. clean esps and create a binary patch for this file. BUT afaik the cleaning process uses all available esps (e.g. skyrim.esp as a master). So you have to be sure ALL esps connected are exactly the same. Then (and only then) you could apply a binary patch: using s.th. like this: http://sourceforge.n...jects/jojodiff/ .
You could also try to automate TES5Edit - no idea about that. Maybe with autoIT?
But both might not be worth the effort...
- renaming doesn't to affect md5 hashes :)
- bsaopt might help with this. Haven't checked it out yet
  • 0

#11 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,250 posts

Posted 03 December 2013 - 01:03 PM

Again, best solution is to contact mod authors and ask them to configure our recommended 'corrections' and reupl;oad and maintain. Thus, the source is 'fixed'.

#12 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 03 December 2013 - 01:46 PM

As has been already stated, nexus meta data simply can't be relied upon. Its just one of the many reasons we don't track mod version. There isn't any way to rely on this data without requiring human intervention.

#13 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,250 posts

Posted 03 December 2013 - 03:22 PM

[quote name=''stoppingby4now' pid='60059' dateline='1386096385']As has been already stated' date=' nexus meta data simply can't be relied upon. Its just one of the many reasons we don't track mod version. There isn't any way to rely on this data without requiring human intervention.[/quote']
This is true, but it may be possible to use file names/dates and parse out the version (which always follows the modID in the filename string), converting it to numeric and using some logic to determine if the package in question has been updated to a higher version #. No 'solution' would be 100%, but it should be possible to net the majority this way ... and those that are ambiguous would require human intervention.

Basically, if MTeg or others are willing to define an algorithm, then it seems possible to reduce the frequency of human intervention so something below 100%. I think that < 25% human intervention may be plausible ... and the actual maintenance would be dependent upon the frequency that the < 25% of these mods are updated.

#14 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 03 December 2013 - 04:31 PM

Yes, if we can get to a point that minimizes the need for human intervention to a small percentage, then it is more viable. Still would need to discuss where to track versions. They aren't a requirement of the Mod pages.

#15 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 03 December 2013 - 04:52 PM

[quote name=''stoppingby4now' pid='60086' dateline='1386106270']Yes' date=' if we can get to a point that minimizes the need for human intervention to a small percentage, then it is more viable. Still would need to discuss where to track versions. They aren't a requirement of the Mod pages.[/quote']
The version info isn't necessary at all. it's just a "nice to have" feature. And it doesn't have to get imported into the wiki at all - no pressure ;)

And yes, thought about the filename -> version parser. But from what I can see, the meta info is doing better than the filenames.
  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users