sarbian

[1.4+ & 1.8+] Hyperspace - Load KSP faster on HDD (or not)

Recommended Posts

29 minutes ago, SquaredSpekz said:

How do you install this? drop the .dll in GameData or the whole zip? Could you please update the folder structure within your download so it's easier to figure where to drop the mod. 

KSP currently doesnt read from .zips
Always extract the contents, which generally have their own folder structure that gets dropped into GameData... ModuleManager is just a dll and its subsequently generated cache files... to install, yes, the .dll just goes straight into /GameData

Share this post


Link to post
Share on other sites

Will this work with KSP 1.3.1--or at least not do anything wrong?  (I'll get around to testing it eventually.)

Share this post


Link to post
Share on other sites
4 hours ago, Jacke said:

Will this work with KSP 1.3.1--or at least not do anything wrong?  (I'll get around to testing it eventually.)

Sarbin said in one of his replies on the first page that it will work back to 1.2.

Share this post


Link to post
Share on other sites

Can it do something with my Win7 64bit 4gb RAM?

Share this post


Link to post
Share on other sites

There's a curious side effect when using HyperSpace: the KSP.log is updated by large chunks of text. It's an annoyance when you live monitor the KSP.log as that last message that you are expecting to check is stuck on the buffer, waiting it to be flushed. But it also ends speeding up the KSP itself, even than just a bit. Writing large chunks of text to the log is faster than writing the same amount of data in small chunks.

So, if you need to live monitor your KSP.log, don't use this and take the hit on the loading time. Other than that, by all means, use this. :) 

On 1/29/2019 at 1:56 PM, Gapone said:

Can it do something with my Win7 64bit 4gb RAM?

Yes. It uses a small extra bit of RAM for each file KSP opens to make reading (and writing) slightly faster. Just a bit faster, but once you have a huge basket of files to be read, that savings pile up.

Go for it.

Share this post


Link to post
Share on other sites

Here are my results.  i7-6820HQ, SSD, 16GB RAM, 81 mods.

[LOG 13:24:40.853] Loading Systems: Elapsed time is 101.4828s
[LOG 13:26:35.238] Loading Systems: Elapsed time is 70.92377s
[LOG 13:28:41.921] Loading Systems: Elapsed time is 69.332s
Hyperspace added
[LOG 13:31:55.804] Loading Systems: Elapsed time is 90.33533s
[LOG 13:33:33.261] Loading Systems: Elapsed time is 67.10857s
[LOG 13:35:11.984] Loading Systems: Elapsed time is 68.48885s

Share this post


Link to post
Share on other sites

So you've swapped the file reader constructor... Have you considered also implementing a way to swap KSP's pseudo-JSON parser?

In combination with GameData Switcher, the game could load compressed files.

 

 

Share this post


Link to post
Share on other sites

Here's my test results:

Plain
[LOG 10:39:22.929] Loading Systems: Elapsed time is 607.2117s
[LOG 10:57:09.827] Loading Systems: Elapsed time is 632.0426s
[LOG 11:11:47.336] Loading Systems: Elapsed time is 472.6618s

After installing Hyperspace
[LOG 11:32:34.623] Loading Systems: Elapsed time is 641.9018s
[LOG 11:48:28.292] Loading Systems: Elapsed time is 471.1871s
[LOG 12:03:03.552] Loading Systems: Elapsed time is 472.1698s

Rig: i7-4770K 3,5GHz, 16GB RAM, Win 8.1 64bit, KSP installed on a Samsung 860 EVO 1TB SSD

CKAN lists 182 Mods installed, plus some additional 30 or so non-ckanned mods and the whole WildBlue Suite.

Share this post


Link to post
Share on other sites
22 hours ago, jeancallisti said:

So you've swapped the file reader constructor... Have you considered also implementing a way to swap KSP's pseudo-JSON parser?

In combination with GameData Switcher, the game could load compressed files.

 

 

Everything is doable but when you do larger change you need increase the risk of needing a large update after each patch. I would rather keep it simple. Compressed file would not really save much loading time. The slowdown is actually after the data is loaded (as shown by the limited impact of this mod)

Some part of the stock loading could be made faster but it s something best done on Squad side... 

 

I think @blowfishlooked at zip loading but I don't remember the results.

 

8 hours ago, LupinoArts said:

Rig: i7-4770K 3,5GHz, 16GB RAM, Win 8.1 64bit, KSP installed on a Samsung 860 EVO 1TB SSD

CKAN lists 182 Mods installed, plus some additional 30 or so non-ckanned mods and the whole WildBlue Suite.

Yeah, the result varies wildly with the hardware. I guess you have enough memory for the whole game to be quickly cached.

Share this post


Link to post
Share on other sites
On 8/17/2019 at 1:10 PM, jeancallisti said:

So you've swapped the file reader constructor... Have you considered also implementing a way to swap KSP's pseudo-JSON parser?

In combination with GameData Switcher, the game could load compressed files.

 

3 hours ago, sarbian said:

Everything is doable but when you do larger change you need increase the risk of needing a large update after each patch. I would rather keep it simple. Compressed file would not really save much loading time. The slowdown is actually after the data is loaded (as shown by the limited impact of this mod)

Some part of the stock loading could be made faster but it s something best done on Squad side... 

I think @blowfishlooked at zip loading but I don't remember the results.

I did.  It ends up being pretty complicated, although I got a working prototype.  It requires reaching deep into KSP's/Unity's/.NET's internals in a way that I'm still not sure I feel comfortable with.  Also some mod plugins do things that are straight up not compatible with it.

Share this post


Link to post
Share on other sites
20 hours ago, sarbian said:

The slowdown is actually after the data is loaded (as shown by the limited impact of this mod)

With the vanilla game, yes, but I also suspect that there's a slowdown when there's a gazillion small files been loaded sequentially, as seen with heavily modded games. Storing those files together in blobs could help speeding up things?

Share this post


Link to post
Share on other sites
48 minutes ago, jeancallisti said:

With the vanilla game, yes, but I also suspect that there's a slowdown when there's a gazillion small files been loaded sequentially, as seen with heavily modded games. Storing those files together in blobs could help speeding up things?

Caveat for this post: I did a lot of testing when I wrote this mod 15 months ago but I do not remember every results. 

The loading (reading from disk and parsing) of the cfg themselves is actually really fast. The slow part is how KSP reads models (and texture?) and build the part DB. The pure disk IO part is not really a big part of it (unless of course if you load the game from a fragmented HDD but not much can help in that case). One of the slow part is how the data stream is divided in byte/long/float/etc 

If you want to understand some of the underlying problem then I refer you to this article that explains what is actually slow (not in a KSP specific context but it is how KSP does it)  

There are some solution in this article than may help with KSP loading but as I said earlier I do not think it is a good idea for a mod to rewrite such a large part of the stock game code. It is something that should be done on Squad side but it takes times away from bugs/features and may not even be a lot faster for everyone...

Share this post


Link to post
Share on other sites
15 hours ago, sarbian said:

The loading (reading from disk and parsing) of the cfg themselves is actually really fast. The pure disk IO part is not really a big part of it. One of the slow part is how the data stream is divided in byte/long/float/etc 

If you want to understand some of the underlying problem then I refer you to this article that explains what is actually slow (not in a KSP specific context but it is how KSP does it) 

Fair enough. Thanks for considering and answering!

Share this post


Link to post
Share on other sites

I am on 1.7.3:

Before Install, cached load time: 189.1024

Run 1 with Hyperspace time: 186.69
Run 2 with Hyperspace time: 138.8109

 

Core i7-7700K @ 4.2Ghz
32GB DDR4 3000
Crucial P1 1TB 3D NAND NVMe PCIe M.2 SSD

Share this post


Link to post
Share on other sites

Amazing!

The loading time went from 586.434 to 434.323!

I dont know how it is done but this is amazing!

Share this post


Link to post
Share on other sites
Posted (edited)
On 4/5/2020 at 1:06 AM, luizsilveira said:

I keep wondering, why isn't something like this in stock by now? Any real down sides?

Because it's NOT ABSOLUTELY improving your performance.

In general, this SHOULD work, but may less stable for all users.

Actually there's some more aggressive method to improving loading speed, but more complex and much more unsafe. I'm currently trying it.

Edited by robinxb

Share this post


Link to post
Share on other sites

I wonder if it needs to be updated for 1.9? Im still using the old copy because anything will do tbh.

Share this post


Link to post
Share on other sites
6 hours ago, Xurkitree said:

I wonder if it needs to be updated for 1.9? Im still using the old copy because anything will do tbh.

I have rebuilt it for 1.8+ to be sure : Download for KSP 1.8+

Share this post


Link to post
Share on other sites
Posted (edited)

Before:

Loading Systems: Elapsed time is 883.7227s

 

After:

Loading Systems: Elapsed time is 602.8457s

 

Intel 5820k@4.4Ghz, 40GB RAM, 2.5GB/s SSD. 

Can rule out caching. Thanks!

 

Modlist:

 

 

000_ClickThroughBlocker
000_FilterExtensions
000_FilterExtensions_Configs
000_TexturesUnlimited
001_ToolbarControl
002_CommunityPartsTitles
999_KSP-Recall
AdjustableModPanel
AECS_Motion_Suppressor
All Tweak
AnimatedDecouplers
ARPIcons
ASET
Astrogator
AstronomersVisualPack
Augustus
AutoLoadGame
AviationLights
B9PartSwitch
B9_Aerospace_ProceduralWings
BDB+ReStockfairingPatch
BetterScienceLabsContinued
Bluedog_DB
BonVoyage
CameraTools
Chatterer
CNAR
Coatl Aerospace
CollisionFXReUpdated
CommNetAntennasConsumptor
CommNetAntennasExtension
CommNetAntennasInfo
CommNetConstellation
CommunityCategoryKit
CommunityResourcePack
CommunityTechTree
ConformalDecals
ConnectedLivingSpace
ContractConfigurator
ContractPacks
Corvus
CraftManager
CrewLight
CrowdSourcedFlags
CryoEngines
CryoEnginesNFAero
CryoTanks
CustomAsteroids
CustomPreLaunchChecks
Decalc'o'mania
DeployableEngines
DE_IVAExtension
DistantObject
DMagicScienceAnimate
DynamicBatteryStorage
EarnYourStripes
EasyVesselSwitch
EngineLightRelit
EnvironmentalVisualEnhancements
EVAEnhancementsContinued
EvaFollower
FairingFix
FerramAerospaceResearch
Firespitter
FlightTracker
FP_DPSoundFX
FuelEfficientPilots
FuelTanksPlus
Fuji
GravityTurn
HabTech
HabTech2
HabTechProps
HGR
HideEmptyTechTreeNodes
Historian
HotSpot
IndicatorLights
IndicatorLightsCommunityExtensions
InternalRCS
InterstellarFuelSwitch
JamesWebb
JetSoundsUpdated
JSI
Jso
KAS
KeR-7
KerbalAtomics
KerbalAtomicsLH2NTRModSupport
KerbalChangeLog
KerbalEngineer
KerbalizedFlagpack
KerbalKonstructs
KerbalOccupationColors
KerbalReusabilityExpansion
KerbinSideRemastered
KGEx
KIS
Klockheed_Martian_CoolRockets
Knes
Kopernicus
KopernicusExpansion
Kronometer
KS3P
KSCExtended
KWRocketry
KXAPI
MagpieMods
MarsDirect
masTerTorch
MechJeb2
MiningExpansion
Mk2Expansion
MOARdV
MOARdVPlus
ModularFlightIntegrator
ModularLaunchPads
ModuleAnimateGenericEffects
ModuleSequentialAnimateGeneric
MSP3000
MunSeeker
NASA_CountDown
NearFutureConstruction
NearFutureElectrical
NearFutureElectricaNTRs
NearFutureExploration
NearFutureLaunchVehicles
NearFutureProps
NearFuturePropulsion
NearFutureSolar
NearFutureSpacecraft
NehemiahInc
Nereid
OPM
OPM_FinalFrontier
OPT
OPT_Reconfig
OSSNTR
PlanetaryBaseInc
PlanetShine
PlanetWiki
Principia
ProceduralFairings
ProceduralFairings-ForEverything
PWBFuelBalancerRestored
RealChute
RealFuels
RealFuels-Stock
RealisticAtmospheres
RealPlume
ReentryParticleEffectRenewed
RemoteTechRedev
REPOSoftTech
ReStock
ReStockPlus
RetractableLiftingSurface
SCANsat
scatterer
SETIgreenhouse
SHED
ShipEffectsContinued
ShipManifest
Sigma
SimpleAdjustableFairings
SituationModuleInfo
SmartParts
SmokeScreen
SolarScience
SolverEngines
SpaceTuxSA
Stock folder: Squad

Stock folder: SquadExpansion

StationPartsExpansionRedux
StationScience
SurfaceLights
SXT
Taerobee
Tantares
TantaresLV
TarsierSpaceTech
TextureReplacer
ThrottleControlledAvionics
ThunderAerospace
ToadicusTools
TrackingLights
TrackingStationEvolved
TriggerTech
TundraExploration
TundraSpaceCenter
TweakableEverything
TweakScale
UAL002
UniversalStorage2
VentralDrill
WarpPlugin
WaypointManager
World-Airforce-Insignia
ZeroMiniAVC
zFinal_FilterExtensions
[x] Science!
999_Scale_Redist.dll
FinalFrontier.dat
Hyperspace.dll
ModuleManager.4.1.3.dll
ModuleManager.Physics
ModuleManager.TechTree
ModuleManagerLicense.md

Edited by Bellabong

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.