Jump to content


Photo

getting the metadata


  • Please log in to reply
32 replies to this topic

#16 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 03 December 2013 - 05:14 PM

[quote='MilesTeg' pid='60087' dateline='1386107540']
[quote name=''stoppingby4now' pid='60086' dateline='1386106270']Yes' date=' if we can get to a point that minimizes the need for human intervention to a small percentage, then it is more viable. Still would need to discuss where to track versions. They aren't a requirement of the Mod pages.[/quote']
The version info isn't necessary at all. it's just a "nice to have" feature. And it doesn't have to get imported into the wiki at all - no pressure ;)

And yes, thought about the filename -> version parser. But from what I can see, the meta info is doing better than the filenames.
[/quote]

One case (and the only one I see as viable) for wanting version number is for testing purposes, so one can track testing results associated with mod changes.

#17 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 03 December 2013 - 05:32 PM

If version number is acquired anywhere and is useful in the ASI process, then it is inherently useful to capture on the wiki. Ideally, the wiki would be the secondary source that is populated (and corrected via some small % human intervention) from the Nexus primary. Thus, all of the Packs and STEP-related ASI input (as well as any future apps) have a reliable source of info. Also, I suspect that the MO regexp (or whatever) that is being used to parse the Nexus data can be improved. There are several cases I have found where MO consistently misses the mark for matching a package to its correct metadata, and the 'fix' is a simple file-name change (usually by removing redundant version chars or the like) ... which means that reason for failure is often predictable, which in turn means that the MO parser could be enhanced to account for this. Perhaps this kind of enhancement could help MO and in turn help ASI ... perhaps we should bring Tannin in here and ask if he could point to the code he uses to parse ... and MTeg et al. could help to improve ... ?

#18 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 03 December 2013 - 06:14 PM

The version number is necessary, unfortunately, because if the user downloads a more up to date version of a mod than what is supported in the config file or the wiki the program won't work. The user needs to be pointed to the file that has the EXACT archive name that is expected. If the mod author doesn't change the name of the archive between updates it won't much matter, but if they do (and most of them do) it matters a lot. If there was some standard, really anything at all, when it came to archive names and mod updates then something could be done to make the program less version specific. Unfortunately there is not. Updating the SAS wiki in real time is unrealistic. The user of ASI would have to be instructed to download older version of mods. At a regular interval, the wiki would be updated to reflect the changes in the STEP mods. The task is encouraging mod authors that don't keep old version of the mod on their page to start doing so. This is an intrinsic property of automation btw, it isn't specific to my program, virtually any algorithm for automation is going to demand reliable input.
  • 0

#19 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 03 December 2013 - 06:17 PM

Perhaps this kind of enhancement could help MO and in turn help ASI ... perhaps we should bring Tannin in here and ask if he could point to the code he uses to parse ... and MTeg et al. could help to improve ... ?

*g* bullseye: versioninfo.cpp

And if I wouldn't have studied regular expressions for half an hour this morning...

QRegExp exp("([0-9]+)(\\.([0-9]+)(\\.([0-9]+))?)?");

there you have it. feel free to fix it ^^

Give me some filenames/version examples and I'll take a look :) (no promises)
  • 0

#20 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 03 December 2013 - 06:29 PM

If there was some standard, really anything at all, when it came to archive names and mod updates then something could be done to make the program less version specific. Unfortunately there is not.


The only thing that might somehow remotely help would be the date attributes from files within the archieves. Just a (very) wild guess...
  • 0

#21 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 03 December 2013 - 08:58 PM

[quote='MilesTeg' pid='60099' dateline='1386113389']
[quote name=''mothergoose729' pid='60094' dateline='1386112443']If there was some standard' date=' really anything at all, when it came to archive names and mod updates then something could be done to make the program less version specific. Unfortunately there is not.[/quote']

The only thing that might somehow remotely help would be the date attributes from files within the archieves. Just a (very) wild guess...[/quote]
There is a general standard: Mod versions usually follow the same naming convention as previous versions, and the version number is increased.

I'll pin down some of the patterns I have noticed in MO file recognition ... unfortunately, I kept no record of the packages I renamed in order to get them to be recognized by MO (there were at least 3). I have more to go, so hopefully, I run into this again. I also noticed that if a mod package does not have a "download with manager" button, it will not be regognized by MO (at least the cases I have found). Also, MO seems to create a partial meta file for mods that it indicates need "query info", so I am not sure whyat exactly causes a mod's meta to be 'complete' enough to register as 'Done'.

#22 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 04 December 2013 - 12:48 AM

Power outages suck! One thing to keep in mind for mod version, is that STEP has historically not cared since everything was a manual process anyway. The biggest problem in relying on version is when mod authors don't keep previous versions available. It's easy to not care when it's just telling a user to download the latest (except for the rarer cases), but something like ASI that relies on a specific versions would need to be able to safely handle cases where a specific version is not available (at least until an update us made). Even then, mod version is not a required trackable at least in relation to the Mod pages. Its an ASI, and testing, requirement.

#23 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 04 December 2013 - 01:36 AM

[quote='z929669' pid='60103' dateline='1386122302']
[quote='MilesTeg' pid='60099' dateline='1386113389']
[quote name=''mothergoose729' pid='60094' dateline='1386112443']If there was some standard' date=' really anything at all, when it came to archive names and mod updates then something could be done to make the program less version specific. Unfortunately there is not.[/quote']

The only thing that might somehow remotely help would be the date attributes from files within the archieves. Just a (very) wild guess...[/quote]
There is a general standard: Mod versions usually follow the same naming convention as previous versions, and the version number is increased.
[/quote]
 There needs to be some kind of general rule that applies to many cases. Many mod authors are consistent with themselves but not with anybody else. Anyway maybe it just needs a great mind to figure it out. You write a workflow or a psuedo code, I can write the C++ :).

[quote='stoppingby4now' pid='60124' dateline='1386136110']Power outages suck!

One thing to keep in mind for mod version, is that STEP has historically not cared since everything was a manual process anyway. The biggest problem in relying on version is when mod authors don't keep previous versions available. It's easy to not care when it's just telling a user to download the latest (except for the rarer cases), but something like ASI that relies on a specific versions would need to be able to safely handle cases where a specific version is not available (at least until an update us made).

Even then, mod version is not a required trackable at least in relation to the Mod pages. Its an ASI, and testing, requirement.[/quote]
SAS doesn't have to change the way regular STEP does things. We will work on it in our own little bubble. If we do a great job, the users will like it, and the project will validate itself. 
  • 0

#24 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 04 December 2013 - 03:14 AM

Perhaps this kind of enhancement could help MO and in turn help ASI ... perhaps we should bring Tannin in here and ask if he could point to the code he uses to parse ... and MTeg et al. could help to improve ... ?

*g* bullseye: versioninfo.cpp

And if I wouldn't have studied regular expressions for half an hour this morning...

QRegExp exp("([0-9]+)(\\.([0-9]+)(\\.([0-9]+))?)?");





there you have it. feel free to fix it ^^

Give me some filenames/version examples and I'll take a look :) (no promises)

The preceding regex will parse out any number from the Nexus file name, which leaves a lot of guesswork to logically sort through. The following is quite a bit more restrictive in that it captures the ID and sometimes other non-decimal number strings, but I have not tested it that thoroughly:
-\K([0-9]+)(?=(-|\.))
or (but this is not quite as restrictive):
-\K([0-9]+)(?:(?!(-|\.)))*

EDIT: I also found an old post where Tannin shared one that will grab the entire file string if that is of any use (but it is only a starting point and will need to be made more flexible without losing any specificity).

#25 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 04 December 2013 - 05:55 AM

I'll generate the core file list. Where we'll get some examples to work with

  • 0

#26 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 04 December 2013 - 05:01 PM

there you go: STEP Core :) Although there are utilities, extender and fixes data missing. Current road block is the nexus maintenance/ MO compatibility with new nexus. But at least it's a start. If you find any errors feel free to report them :) There are probably a lot more files missing, cause I didn't get to download all versions / complex mods for now. Spreadsheet data: https://drive.google...dit?usp=sharing
  • 0

#27 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 04 December 2013 - 05:29 PM

Damn near a config file. Good start keep it up :)
  • 0

#28 MilesTeg

MilesTeg

    Citizen

  • Members
  • Pip
  • 91 posts

Posted 04 December 2013 - 06:26 PM

Damn near a config file. Good start keep it up :)

nah, doesn't handle any complex archives, yet....

this one, on the other hand :lol:

edit: I've also some cool ideas of how we can smoothly integrate the STEP mods in MO. There will be mod descriptions, automatic install order etc...

Attached Files


  • 0

#29 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 04 December 2013 - 09:21 PM

That is the CRC key for each file and its relative path? That is pretty much half the battle there. Just need to write the bit that finds the same files in source archives. Matching them up after that would be a peace of cake. And yeah, emulating a MO install or a regular skyrim install, either would work with that approach. Chomping at the bit to get at that stuff... couple more weeks and I will have time to code again. Great stuff keep it up.
  • 0

#30 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 05 December 2013 - 02:33 AM

I'll generate the core file list. Where we'll get some examples to work with

Yep, my revised regex works seemingly better on this entire list than what Tannin is currently using in MO (assuming that more restrictive matches on numbers representing NexusID is more desirable. My version cuts down on non-ID-number matches by about 30-40% I think without loss of fidelity on the above list.

Also, I have improved the Core mod CSV export SMW query (which, incidentally, was limited to 100 returned records ... the limit= argument does not work, so I edited the PHP source code to 1000 as a temporary hack :/ ). The new query gets all relevant 135 Core mods that can be installed with ASI (I assume that Tech went in and fixed mods marked as Core that were not in current v2.2.7 ... thanks!), less the utilities and extenders, which are manual installs.
{{#ask: [[Category:Mods]][[IsCore::true]][[Section::!C]][[Section::!B]][[Section::!A]]
| Mainlabel=-
| ?Section=Section
| ?=Short Name
| ?FullName=Full Name
| ?SourceID=NexusID
| sort=Section,FullName
| format=csv
}}



0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users