Help with KVV - Material Shader to string Deprecated

[bigorangemachine]

Hi All,

Since the Unity5 update KVV (Kronal Vessel Viewer) has only been rendering Pink Screenshots.

I went through and managed to figure that this line is part of the issues I'm having.

I'm mostly a Web Developer who is pretty accomplished as just hacking through existing code(s) and understand it enough to 'be dangerous' but when it comes to shaders I'm pretty lost. I have some 3D background but never got deep into shaders/materials.

 

So in short I'm wondering how to get around this unity5 issue.  In short KVV is reading the edge detection shaders and then using a regexp to adjust values.  However I'm not sure how shader logic works nor how I would re-import a shader after its been converted from a string back into a material.

Any help/insights will be greatly appreciated

[Andem]

I've been using the old KVV version and haven't had any issues with the screenshots, but good luck with updating!

[MOARdV]

On 5/5/2016 at 0:16 PM, bigorangemachine said:

Hi All,

Since the Unity5 update KVV (Kronal Vessel Viewer) has only been rendering Pink Screenshots.

I went through and managed to figure that this line is part of the issues I'm having.

I'm mostly a Web Developer who is pretty accomplished as just hacking through existing code(s) and understand it enough to 'be dangerous' but when it comes to shaders I'm pretty lost. I have some 3D background but never got deep into shaders/materials.

 

So in short I'm wondering how to get around this unity5 issue.  In short KVV is reading the edge detection shaders and then using a regexp to adjust values.  However I'm not sure how shader logic works nor how I would re-import a shader after its been converted from a string back into a material.

Any help/insights will be greatly appreciated

I'm far from a Unity expert, so take this as my understanding of the issue, which may be incomplete.  The problem line (the one generating the warning) is just below the one you pointed at, where it has

this.Material = new Material(contents);

Unity 5 deprecated loading shaders from strings, which is what this line is doing.  What you have to do instead is create an AssetBundle using Unity and KSP's PartTools 1.1 (creating the AssetBundle was painful until I edited the bundle's XML file by hand), and then have the mod load the shader from the AssetBundle at runtime.  It looks like the mod uses multiple shaders, so you'd have to do something similar to what I did in RasterPropMonitor to load them (see this class for my solution).  Once they're loaded into a dictionary or whatever, you fetch the shaders and pass that into 'ShaderMaterial' as a parameter instead of a string.

As for the pink screenshots - that's Unity's way of saying that the shader did not load / compile correctly.  I've had it reported in KSP 1.1.2 for people using -force-glcore in Windows.  I don't know what the problem is with that.  It is possible that the pink shader you're seeing has nothing to do with the warning - all of the shaders in RPM still worked in KSP 1,1.x using the old way of loading them.

[wasml]

@MOARdV Thanks for the Asset loading example - I've been tearing my hair out trying to figure out why I couldn't load my three assets - thought I was doing something wrong.

[bigorangemachine]

On 07/05/2016 at 11:09 AM, MOARdV said:

I'm far from a Unity expert, so take this as my understanding of the issue, which may be incomplete.  The problem line (the one generating the warning) is just below the one you pointed at, where it has...

Hey thanks. So do I have to unpack the shader using the asset bundler as well? As i understand the code there is a regular expression modifying the exisitng shader. Are shaders still able to convert to a string?

[MOARdV]

2 minutes ago, bigorangemachine said:

Hey thanks. So do I have to unpack the shader using the asset bundler as well? As i understand the code there is a regular expression modifying the exisitng shader. Are shaders still able to convert to a string?

Oh, right.  I didn't look that closely what the regex was doing.  It looks like it's searching the properties in the shader to build a table of values/properties it can modify.  The Unity Material class can change the values, and it can query if a particular name exists, but I don't know if it has a way to give you a list of all properties (there is a shaderKeywords field, but I haven't looked at it, so I don't know what it contains).  I don't know if there's a way to fetch the shader text from the Shader class, and I don't know for sure if there's a way to get properties out of the Material in a way you can use.  Sorry, that's outside my limited Unity expertise.

[FancyRaptor]

Hey is it possible that once you guys get it working you could put a download up for a working version for 1.1.2. I cant even get the KVV icon in the bottom

[sarbian]

10 hours ago, FancyRaptor said:

Hey is it possible that once you guys get it working you could put a download up for a working version for 1.1.2. I cant even get the KVV icon in the bottom

No. When something work we delete it, hide the source in a safe and dump it in the mariana trench.

 

As for the original problem : you did recompile the shader and you load the compiled version right ? Because loading from a string still works (I use it in MJ). It will be deprecated in a future version.

Edit : and I fail to understand the point of this code. Why generate a list of the shader properties ? If you write your shader you should know what each prop do and if you load a arbitrary shader you can't know what each prop do...

Edited May 9, 2016 by sarbian

[bigorangemachine]

2 hours ago, sarbian said:

No. When something work we delete it, hide the source in a safe and dump it in the mariana trench.

 

As for the original problem : you did recompile the shader and you load the compiled version right ? Because loading from a string still works (I use it in MJ). It will be deprecated in a future version.

Edit : and I fail to understand the point of this code. Why generate a list of the shader properties ? If you write your shader you should know what each prop do and if you load a arbitrary shader you can't know what each prop do...

I think I get what you are saying. I was worried about the longevity of the fix (believe me I would be very happy just patching it but all my projects are biting me in the @$$ right now where I cut corners like this).

So there is a way to iterate over these properties using some kind of 'iterator'? So Kronal just 'chopped out' the parts he didn't like rather than change the values inside an object.  I'm not sure why Kronal wrote it the way he did; but we are modifying shaders we know nothing about (other part packs).  Like I said in the OP; I don't know shaders so your last question is a lot of greek to me lol

I got a lot (for me) to look into but it doesn't seem as big as a deal as I first thought

 

Which part of MJ are you messing with shaders (I don't use MJ I am not aware)

Edited May 9, 2016 by bigorangemachine

[sarbian]

12 minutes ago, bigorangemachine said:

So there is a way to iterate over these properties using some kind of 'iterator'? So Kronal just 'chopped out' the parts he didn't like rather than change the values inside an object.  I'm not sure why Kronal wrote it the way he did; but we are modifying shaders we know nothing about (other part packs).  Like I said in the OP; I don't know shaders so your last question is a lot of greek to me lol

I ll have to look at how this work more because AFAIK you can't get a shader source from the game so the only shader you will parse are those included with the mod.

 

12 minutes ago, bigorangemachine said:

Which part of MJ are you messing with shaders (I don't use MJ I am not aware)

I needed a simple shader that ignore the z buffer to display the center of mass and by debugging arrow even when they are inside a part, and with some lighting change to not have them look too flat.

[rspeed]

On 5/7/2016 at 11:09 AM, MOARdV said:

I've had it reported in KSP 1.1.2 for people using -force-glcore in Windows.

Seems plausible that the issue only occurs when using Unity's OpenGL Core. I've been having the issue on OS X, which uses OpenGL, but not in Windows when using the default DirectX.

Edited May 9, 2016 by rspeed
Better wording

[Van Disaster]

Loading from string seems erratic for Mac users ( B9-pw had that problem ) - loading more than one shader from a bundle is also buggy unless you ignore the supplied bundle-loader, however it's not hard to knock up an asset loader ( there's a recent thread about that here which I'll dig up when I find it again - I've written code to load & replace shaders but you don't want to be using my atrocious beginner code as an example! ). Edit:

and check the source to IR as suggested.

Technically you can just iterate materials, get the existing shader name ( or anything else you want to use as a key ) from the current material & replace the shader reference in the material with a new one; I'm still working through the wrinkles of doing that myself...

Edited May 19, 2016 by Van Disaster

[bigorangemachine]

On 5/8/2016 at 0:11 PM, MOARdV said:

Oh, right.  I didn't look that closely what the regex was doing.  It looks like it's searching the properties in the shader to build a table of values/properties it can modify.  The Unity Material class can change the values, and it can query if a particular name exists, but I don't know if it has a way to give you a list of all properties (there is a shaderKeywords field, but I haven't looked at it, so I don't know what it contains).  I don't know if there's a way to fetch the shader text from the Shader class, and I don't know for sure if there's a way to get properties out of the Material in a way you can use.  Sorry, that's outside my limited Unity expertise.

 

On 5/19/2016 at 11:59 AM, Van Disaster said:

Loading from string seems erratic for Mac users ( B9-pw had that problem ) - loading more than one shader from a bundle is also buggy unless you ignore the supplied bundle-loader, however it's not hard to knock up an asset loader ( there's a recent thread about that here which I'll dig up when I find it again - I've written code to load & replace shaders but you don't want to be using my atrocious beginner code as an example! ). Edit:

and check the source to IR as suggested.

Technically you can just iterate materials, get the existing shader name ( or anything else you want to use as a key ) from the current material & replace the shader reference in the material with a new one; I'm still working through the wrinkles of doing that myself...

@MOARdV how do you get the XML into the C#?  I see you have an xml file; but I don't see any reference in your code to that XML.  I am new to C# (not formally trained) some of the wizardry gets lots on me.

[MOARdV]

54 minutes ago, bigorangemachine said:

 

@MOARdV how do you get the XML into the C#?  I see you have an xml file; but I don't see any reference in your code to that XML.  I am new to C# (not formally trained) some of the wizardry gets lots on me.

You don't.  The XML file is a manifest that PartTools uses to assemble the AssetBundle.  Once it's in KSP, you need to use KSP's AssetLoader (or a workaround like I did) to fetch the Shaders.  I know what my asset bundle is called, so I refer to that when I ask the AssetLoader to find the shaders.

[bigorangemachine]

24 minutes ago, MOARdV said:

You don't.  The XML file is a manifest that PartTools uses to assemble the AssetBundle.  Once it's in KSP, you need to use KSP's AssetLoader (or a workaround like I did) to fetch the Shaders.  I know what my asset bundle is called, so I refer to that when I ask the AssetLoader to find the shaders.

Ah thanks! I don't have part tools

[bigorangemachine]

@MOARdV,

I got Unity to generate a .ksp file with some test shaders.  For some reason I keep getting

 

Quote

AssetLoader: Cannot find bundle definition with name 'kvv'

 

Tried both

KSPAssets.AssetDefinition[] KVrShaders = KSPAssets.Loaders.AssetLoader.GetAssetDefinitionsWithType("kvv", typeof(Shader));

and

KSPAssets.AssetDefinition[] KVrShaders = KSPAssets.Loaders.AssetLoader.GetAssetDefinitionsWithType("KronalUtils/kvv", typeof(Shader));

I've uploaded what I have here to github.

 

I'm sure I've done something simple wrong.

 

I must be doing something simple wrong

[MOARdV]

3 hours ago, bigorangemachine said:

I'm sure I've done something simple wrong.

It's not really straightforward, and the readme leaves a little to be desired, so don't be discouraged.  For RasterPropMonitor, I do this:

KSPAssets.AssetDefinition[] rpmShaders = KSPAssets.Loaders.AssetLoader.GetAssetDefinitionsWithType("JSI/RasterPropMonitor/rasterpropmonitor", typeof(Shader));

That's the entire path to the Asset Bundle from Gamedata, with the name of the bundle (minus the .ksp extension).  So, presumably your asset bundle is named kvv.ksp, and it's in the KronalUtils directory.

Are there any shaders in the asset bundle?  The file looks like small on GitHub.  I suspect there aren't any shaders in it.  You have to find the shaders in the project window in Unity 5, click on them, and make sure the Asset Labels window in the lower-right corner has "kvv" in it.  Take a look at the screen shot sarbian posted to see what I mean.  If it is visible there, I'd also check to make sure the shader compiled successfully.

[bigorangemachine]

Thanks for the quick reply @MOARdV!

 

The KSP file isn't empty; I checked the bundle XML and the shaders are listed there.  The 2 shaders I'm currently migrating are there and they are only about 6 lines.  So it makes sense its small.  The contents actually match what I had archived (before posting here) and what was rendered after I verified against the bundle XML.  

 

In unity I noticed that when I tried to create the asset bundle under 'KronalUtils/kvv' unity complains that I don't have an '~/Unity Projects/KVV/Assets/kronalutils' folder (even after I created it).  I'm thinking maybe that path is more about how it gets declared in the bundle.xml rather than path inside the GameData/ directory?

[MOARdV]

4 hours ago, bigorangemachine said:

In unity I noticed that when I tried to create the asset bundle under 'KronalUtils/kvv' unity complains that I don't have an '~/Unity Projects/KVV/Assets/kronalutils' folder (even after I created it).  I'm thinking maybe that path is more about how it gets declared in the bundle.xml rather than path inside the GameData/ directory?

'~/Unity Projects/KVV/Assets/kronalutils'? It's not mismatched case in the path name, is it?  Linux tends to be fussy about that.

My bundle lists assets with a shorter relative path:

<Asset Name="RPM/FontShader" Path="Assets/RasterPropMonitor/Shaders/RPM-FontShader.shader" Type="Shader" Url="RasterPropMonitor/Shaders/RPM-FontShader" AutoLoad="false" />

But I'm also on Windows.  The "Assets" directory is the Assets directory that's part of the Unity 5 workspace.  Squadcore is one of the other directories underneath Assets, so I assumed I should follow their lead for organizing files.

I'm afraid I'm not sure what's going on in your case.  Let me download your shaders and try building the bundle with them here.

[bigorangemachine]

Oh I only meant it as in "This stuff is in my home directory". I'm on windows 8.1.

 

I'm wondering if there is some square bracket stuff I'm missing.  I'm new to C# and I'm not entirely sure what they mean.  Is that something I'm missing?

[MOARdV]

5 minutes ago, bigorangemachine said:

Oh I only meant it as in "This stuff is in my home directory". I'm on windows 8.1.

 

I'm wondering if there is some square bracket stuff I'm missing.  I'm new to C# and I'm not entirely sure what they mean.  Is that something I'm missing?

Ah, okay, no Linux-stuff going on then.

I was able to create the asset bundle and load it in game using the two shaders on your GitHub site.  As for loading them, take a look at the code in RasterPropMonitor (specifically, UtilityFunctions.cs, line ~1695, which would be something like

KSPAssets.AssetDefinition[] kvvShaders = KSPAssets.Loaders.AssetLoader.GetAssetDefinitionsWithType("KronalUtils/kvv", typeof(Shader));

in your case (assuming you put the kvv.ksp asset bundle in GameData/KronalUtils).  After you make sure kvvShaders.Length > 0, you would do this (from line ~1780 in the same file)

KSPAssets.Loaders.AssetLoader.LoadAssets(AssetsLoaded, kvvShaders[0]);

AssetsLoaded is the function that takes those shaders and puts them somewhere useful - it's defined at and below line 1985 in UtilityFunctions.cs.  The code is somewhat ugly because KSP's asset loader doesn't seem to handle loading more than one shader from an asset bundle (unless they fixed that in 1.1.3), so we have to identify one of the shaders in the bundle we got through the callback, and we use that to find the asset bundle in KSP's list of loaded bundles, so we can find the rest of our shaders.  Like I said, it's kind of ugly code.

 

[bigorangemachine]

Hey Thanks again

I tried variations like that.  It always says it can't find my assets.  I've tried every variation I can think of.

Right now I'm trying to go with a core unity bundle; having issues there too

Keep getting 

This AssetBundle was not created with UncompressedAssetBundle flag, expected id 'UnityRaw', got 'UnityFS'

when built using

BuildPipeline.BuildAssetBundles(outputPath, BuildAssetBundleOptions.UncompressedAssetBundle);

Ugh.. can't win

[MOARdV]

Yeah, KSP (fortunately) compresses the asset bundle.  Unfortunately, Unity can't handle that compressed format, so you're forced to go through the KSP mechanism.  I don't recall that there's a "don't compress this" option for the asset bundle that might get you around that.

[bigorangemachine]

I think its an editor bug (Build tools). I posted in the unity forums to see if i can get help there.

 

So you were able to KSP-bundle with my shaders and load them in? I see where the code is failing; i can't the bundle to load/found even copying yours.

[sarbian]

Just to be sure : you made your bundle with Unity 5.2.4 ?