KerbalX.com - Craft & Mission Sharing

[Val]

Or you could make it dynamic. Load, store, crop, resize it, the first time a user needs to see a thumbnail with that image.

Edited February 13, 2016 by Val
Clarified

[Mad Rocket Scientist]

On 2/12/2016 at 7:20 PM, katateochi said:

Done!! https://kerbalx.com/statistics 
(that page was there before, but kinda horrible and didn't really show much, been meaning to update it for a while)

[...]
 

That's a great page!  I didn't know I was the 6th uploader.  And I've got 7395 downloads!

[katateochi]

Big shout out to @Raptor9 for just sending a donation!! Thanks Dude!! 

2 hours ago, Val said:

Or you could make it dynamic. Load, store, crop, resize it, the first time a user needs to see a thumbnail with that image.

That would cause quite a big slowdown on the site as typically it would be ~30 thumbnails per go and that would be on the main server thread.  But....hang on, this answer fits in with the next bit, see below;

@swjr-swis yeah a bulk process of all the craft would cause problems. but I don't think that it's going to be as bad as I thought to make the thumbnails for existing craft as I thought.  
As much as I want to write a multi-threaded script to just hammer imgur hard, I shall resist.  So....

Going forwards; new craft will work exactly as they do now; user adds images to page, then chooses one (or site selects the first one if none selected) to be thumbnail.  The difference is that when they hit save that image gets downloaded and the thumbnail is generated and stored.  I think I can make it so that if you didn't know anything was different you won't even notice (initial save might be a bit slower).  This, at least from my reading of their emails, is ok use of imgur.  They say a user can post an image and I can generate a thumbnail from it, so logically there is no other way.

Going backwards; to get the thumbs for existing craft, I will batch process the craft in groups of like 10 at a time, with a bit of a delay between batches and I'll do that on KX's background worker queue system.  The worker queue is like a separate instance of the server which runs in the background so it won't slow the site down at all while doing that.  And as far as they are concerned.....

Spoiler

What's nice is that it should be a fairly graceful change over.  Craft that have the current thumbnails will continue showing that image url (and as we're now on https, imgur hasn't cottoned on so they still show) and as they get the thumb generated they'll just update their url.  Now I've said that it's all going to explode and I'm going to be freaking out like.....well, Bob during reentry! 

I won't have the upload of images to KerbalX feature done by this weekend.  That requires a bit more interface work (which I'm much slower at).  But the thumbnail stuff should be working by tomorrow.  It's basically working in development now, but needs refining, tests writing and some other stuff.  Prob won't deploy till Monday morning though (usually lower traffic in the AM UK time). 

[katateochi]

Ok right, well.....I think I'm ready to push changes up to the site! I've just put a downtime alert up, but I probably won't have to take it offline, but things might go weird!
Uploading craft will be disabled for a bit while I get things setup.

ok so, wow, setting up the image hosting storage, writing the code to connect to it, and bits to do image resizing and cropping, that bit was easy.  But what I spent most of yesterday utterly confused about was the more web adminy things of setting up CNAME records and CORS rules (didn't even know about them 'till yesterday) so images will also get handled by the CDN (content delivery network) that I've setup (didn't know much about them either).  I think I've got it set so the CDN will cache and provide better access speeds to geographic locations further from the image storage servers, but I'm not 100% sure.  So image loading might be slower initially while I continue to figure that out.

I still have more to learn about that; for now thumbnails will have urls like s3.amazonaws.com/thumbs-dev.kx/9860.png  (the number being the craft's ID), which is ok. but I still need to figure out how to set it up for the main image hosting so the image urls will be something like i.kerbalx.com/<some_uniq_key>.format (I think I understand that, kinda, now). But I can get this running first and then spend more time pondering that. 

 

[adsii1970]

@katateochi - With the announcement that Kerbal Stuff is shutting down, what is the possibilities that mod hosting could be added to the site? I think it could really be good for KerbalX, and I know that I, for one, would be willing to contribute more financially to support such and endeavor...

[swjr-swis]

1 hour ago, adsii1970 said:

@katateochi - With the announcement that Kerbal Stuff is shutting down, what is the possibilities that mod hosting could be added to the site? I think it could really be good for KerbalX, and I know that I, for one, would be willing to contribute more financially to support such and endeavor...

That's a pretty hefty commitment. 61GB worth of mods, which is likely to increase significantly as soon as KSP 1.1 hits and a large number of mods have to be updated. $180/mo for hosting, according to @SirCmpwn.

Don't get me wrong, I'm not adverse to the idea, I would love to see it crafts and mods integrated in one site, and katateochi has won my confidence, but this is not something to take lightly. I really don't want to see katateochi end up the next burned out a year down the line, and financially it would need more than a dozen or so contributors to help keep it afloat. I'll be supporting too if it comes to it, but please think very seriously before taking it on.

 

As a parallel exercise, I have also messaged Dark0ne of the nexusmods.com site on this. The Nexus has a proven track record as one of the mayor mod hosting sites out there, and he already has both infrastructure and personnel to support it - seems that KSP would hardly make a dent in his current operation. What do you guys think?

[katateochi]

oh wow! damn! ok, hang on, first;

KerbalX is up (ages ago actually, just forgot to post) and all's looking good with the thumbnails.  Should be able to get imgur off my back soon! yay.  
For now it's the same thing for users, no on site image hosting as yet, but that is still coming. I just want to spend more time on making that work nicely, this had to be done in a rush.  Lean't a lot though!! 

Which brings me on to the next thing, but first. A moment of silence for KerbalStuff...........

 

That's really sad, what's even sadder is people all pointing fingers in all directions when really it's the community that's taken the Jam out of @SirCmpwn's doughnut.

So hmmm, could I host it? well, I've just learn't all these cool stuff-ur-data-in-amazon tricks and that would certainly be the way forward I think.  Bassically everything runs off AmazonAWS, netflix, imgur, NASA's JPL, KerbalX (just the important ones  ), it has the hardware to deal with the scale. But what are the costs? 
Well, I haven't got enough data to make a realistic and meaningful prediction but taking some wild guesstimates;

If it was 80GB of data to be stored with 10,000 PUT/POST requests per month, 500000 GET requests and data transfers of around 500GB per month, apparently (and I can't actually believe this), that would come to $55 per month! so what are we waiting for? ans: the stuff we don't know yet! 

That would be a setup which is no different to what I'm now using for the thumbnails and my first reaction when setting that up was hmmm, those load kinda slowly.  A quick google and I found I needed to setup a CDN to help spread the load.  Amazon have a CDN service and from what I've heard it's horribly expensive. I'm using a CloudFlare and on the entry tier, but I don't think that entry tier would be Jeb enough for the job.  So there are more hidden costs and I don't want to jump in blindly.  There's also domain name registrations costs, yada yada....

IF (<big if) I was to attempt this, I I'd keep it and KerbalX as separate services.  My goal would be to litterally bring KerbalStuff back up exactly as it was.  but I don't even know what it is written in (my focus is rails, been a long time since I worked in anything else), but that's not such a problem.  

Anyway, that's my thoughts on it.  Initially I was like, noooooope, couldn't possibly run that. but thinking about it more, it looks less impossible.  But currently; I'm not saying I will or I won't.....I'm pondering it. Any more detailed info on what the traffic would actually be like would be helpful.  I will try and contact @SirCmpwn and find out more, but having just got the image stuff on KX sorted (and having wasted so much time chasing untamed ornithoids with imgur) I kinda need to give work a bit more focus.  But if something is going to happen on this front I think it needs to happen pretty quickly. 

 

EDIT - kinda just jumped into that, sorry bit of a unfiltered brain-to-keyboard post.

@swjr-swis yeah the costs could run up to that even if the initial amazon hosting looks reasonable.  I certainly couldn't afford that and even with you guys helping it's a bit of a risky place to be in unless the site generates it's own revenue.  Also, yes nexus does look like a good solution (it just won't have the great CKAN integrations that KS had).  

Edited February 15, 2016 by katateochi

[katateochi]

ok looks like someone else has picked up the baton and legged off with it - http://spacedock.info/ Don't known who the mystery person(s) who are trying to get it back up is, any ideas?

[swjr-swis]

4 minutes ago, katateochi said:

ok looks like someone else has picked up the baton and legged off with it - http://spacedock.info/ Don't known who the mystery person(s) who are trying to get it back up is, any ideas?

I don't know. I tried checking the reddit thread but no one really owned up to it, and I do not have the bandwidth right now to go monitor the IRC channels they mention. So I'm just gonna sit by the lines and watch the development. I don't reuse passwords for websites, so I'm running next to 0 risk, but I'd still want to see some very clear and open statement about who is behind that new site, before I go for it.

 

I wish I had noticed @SirCmpwn's words in the kerbalstuff thread two months ago; I was completely oblivious to his struggle, I could've jumped in with some support then. I really loathe the reactions of some people at the moment, as if they have been taken something they were somehow entitled to. Sure, I too wish things had transpired different, but in the end, it is/was his ball to take home for whatever reasons (or even without any reason at all for that matter). In fact, he didn't even really take his ball home - he has left the entire site source and the mod repository available, as a last respect to the open source nature of the site, which is more than others have done in the past.

He's got my respect for the effort and time spent in the past years keeping it up and running, and my apologies for having taken it for granted all this time, so even thought I know it makes no difference to the current situation, I punted him some token funds to his paypal - at least he can have a sip or two of his favourite beverage to conclude this chapter. Cheers @SirCmpwn, and thanks.

[RYU AZUKU99]

So I just saw that Kerbal Stuff is being taken down after so long! Please tell me that you arent planning on heading that way as well!

[katateochi]

50 minutes ago, RYU AZUKU99 said:

So I just saw that Kerbal Stuff is being taken down after so long! Please tell me that you arent planning on heading that way as well!

Heck no!! KerbalX isn't going anywhere.  And....if I did finally get fed up with it, I would pass it on to someone first.  But that ain't going to happen, you guys have been awesome with encouragement and I love making this site!

[RYU AZUKU99]

9 minutes ago, katateochi said:

Heck no!! KerbalX isn't going anywhere.  And....if I did finally get fed up with it, I would pass it on to someone first.  But that ain't going to happen, you guys have been awesome with encouragement and I love making this site!

Oh thank god! Thanks for keeping it going and good luck farther down the line. And, just out of curiosity, what are your thoughts on people trying to resurrect the KerbalStuff website.

[adsii1970]

23 minutes ago, katateochi said:

Heck no!! KerbalX isn't going anywhere.  And....if I did finally get fed up with it, I would pass it on to someone first.  But that ain't going to happen, you guys have been awesome with encouragement and I love making this site!

You better not... I finally figured out how to do the PayPal monthly donation bit...

[katateochi]

1 hour ago, RYU AZUKU99 said:

And, just out of curiosity, what are your thoughts on people trying to resurrect the KerbalStuff website.

I'm impressed at the speed of the response to bring it back up.  KS was always my preferred place to get mods and if they can resurrect it and make it more reliable then overall the community comes out ahead.  I'm just a bit concerned about some of what's posted on the thread about how CKAN will work now, as that does have an impact on KerbalX (not to mention playing KSP!).  But for now we just need to wait and see what happens in the next week.

lol, my guestimate of 500GB data transfer per month was waaaay off, apparently it's more like 10-20TB per month! That would put the price up much more for AWS hosting.

[katateochi]

On 12/02/2016 at 2:44 PM, swjr-swis said:

The struts count must be a lie!

so I just did the correct count and it's 116798 struts in total! That seems more like it! 
But there's a problem, after the initial upload I minimize how often the actual craft file needs to be fetched from the database, so during the initial upload various bits of info are stored on the craft object as separate attributes. total part count is one of these, but not strut count.  So to get this real count I have to load every single craft, read their files, pull out all part names and then select those which match names of strut parts.  On my machine that operation took about 5 minutes to run and consumed 10GB of ram!! So I kinda can't do that on the site, not even as a once a month thing.  But now I'm set on having that count, so either I'll add a strut count attr to craft, but then I have to back process all existing craft again. Or I'll add a counter to the main System object which gets updated as craft are added/deleted and initialize it with a count that I'll do of a fresh clone of the production DB when I come to deploy that.  Why is it the "simple" ideas are never simple and things that seem really tricky just work?

 

We're getting there with getting all the thumbnails processed.  ~4000 down, ~2000 more to go! 
Have you guys noticed that thumbs are now cropped rather than squashed if the original image was v wide or tall?  Previously I just had to make the image fit, now I can crop it around the center so it's the right dimensions.  So you shouldn't see anymore distorted thumbs! yay!
 

[swjr-swis]

8 hours ago, katateochi said:

so I just did the correct count and it's 116798 struts in total! That seems more like it!

I knew it! My trust in Kerbal engineering has been restored.

 

8 hours ago, katateochi said:

On my machine that operation took about 5 minutes to run and consumed 10GB of ram!! So I kinda can't do that on the site, not even as a once a month thing.

Repeat after me: It is ok to not have 99.999% uptime. Seriously. It's ok. Maintenance moments are necessary in every server environment, and we're not talking about a critical service here (but I need to upload my craft rite nao!!!!). Just give a head's up on the front page, or even here in the thread, and run the stuff that needs heavy processing during those windows.

 

8 hours ago, katateochi said:

Have you guys noticed that thumbs are now cropped rather than squashed if the original image was v wide or tall?  Previously I just had to make the image fit, now I can crop it around the center so it's the right dimensions.

I did notice, in fact I did a double take on a few craft because it seemed like the the thumbnail was 'not right'... actually now it is right and it was distorted before.

Another thing I noticed, not sure if entirely intended: the two placeholder thumbnails are now squashed into the top half of the thumbnail space.

[katateochi]

3 minutes ago, swjr-swis said:

Repeat after me: It is ok to not have 99.999% uptime. Seriously. It's ok. Maintenance moments are necessary

yeah 99.999% is tricky.  KX has an uptime of 99.779% (over the last 3 months, which is as far back as NewRelic shows me)
but even if I was to have a downtime to run that calc, it still wouldn't be able to run on the server, I just don't have the RAM available.  I could do it with much lower ram requirements by selecting each craft from the DB one at time, but then it would take even longer to run.  I'd make it a background task too so I wouldn't need to go offline (or even slow the site), but a background task running for a long time isn't ideal.  I think there's a more elegant solution.

 

12 minutes ago, swjr-swis said:

Another thing I noticed, not sure if entirely intended: the two placeholder thumbnails are now squashed into the top half of the thumbnail space.

hmmm, that's not intended. Can you show me a screenshot of that cos they look ok to me. (also what browser are you on)

oh and I know that on Internet Explorer there are several things which look horribly wrong.  Will fix those soon. 

 

I may also have goofed a bit on the thumbnails.  I resized them to fit the size they are need on the main results page.  But on pages like hangars they are displayed larger so the thumbnail looks a little low quality.  But I think I'm going to have to live with that for the existing craft (not reprocessing them all again!).  On new craft I'm going to increase the resolution of the thumbs a bit.  I could do it two ways; the tool I'm using allows me to store several versions of the same image (ie at different sizes), so do I store two versions at different res (which more than doubles the storage overhead) or do I just store the high res (which means higher overhead in data transfer, but not double).  I can't decide which to do currently. 

I'm still planning to do full image hosting, but I have decided to delay doing that for a bit.  I want to see how traffic levels and costs work out with just the thumbnails which will give me a better understanding of what the costs will be.  We've already ripped through the free allowance for this month (but with the high numbers of uploads while generating thumbs for everything that was expected).  I need to let it settle out and see what baseline traffic (for GETS) is, multiply that by the average number of pics per craft and see what that looks like.  I'm setting up a patreon account too, and once I have a good understand of the image hosting cost I will set that as a goal and if that's reached then I'll go for it.  
Sorry to be changing course on that, I got a bit carried away with what looked possible! But I don't want to back myself into a place where I can't afford to keep certain features running (certain recent events have been a good reminder to only byte of what I can chew!)

I also want to spend some time refining, nay, fixing some borked things.  The new edit interface does some odd things, especially when adding new images (nothing to do with recent image stuff, I'd noticed that before) so I want to get that sorted.  And there's a couple other fixes on my list I want to check off.
I also want to add more support for detecting partless mods and also try to find ways to improve detection (or rather un-detection) of TweakableEverything and FAR.  Adding support for partless mods is a bit painful (partly why it keeps getting pushed back).  I have to get the mod, stick in in KSP and then make a craft that uses it and then dig around in the craft file and figure out what unique things it adds which I can detect and if it has anything like TS and KerbPaint which enables me to ignore it if present but not active.
I could use some help on that font; telling me which partless mods are the most important ones to add support for would be very useful.  Even more useful is if you know how that mod alters the craft file and can tell me what to look for and what (if any) the ignore clause would be. (like @pellinor did in this post)

 

[swjr-swis]

1 hour ago, katateochi said:

hmmm, that's not intended. Can you show me a screenshot of that cos they look ok to me. (also what browser are you on)

Well, now it's showing ok for me too, so not sure what happened there. Using Firefox 44.0.2 at the moment, although I'm on the hunt for something else, forced by dumbwit decisions by the current FF dev team (ie. imposing their view of how browsing should work on everyone rather than allowing knowledgeable users the options to opt out and/or customize their own experience).

 

1 hour ago, katateochi said:

I want to see how traffic levels and costs work out with just the thumbnails which will give me a better understanding of what the costs will be.

Wise decision, got my support.

 

1 hour ago, katateochi said:

I could use some help on that font; telling me which partless mods are the most important ones to add support for would be very useful.  Even more useful is if you know how that mod alters the craft file and can tell me what to look for and what (if any) the ignore clause would be.

It's a busy week ahead, but I'll try find some bandwidth to go through a few test cases. Do you have a priority list of mods you want checked first?

[katateochi]

3 minutes ago, swjr-swis said:

Well, now it's showing ok for me too, so not sure what happened there.

There's a little bit of javascript that attempts to make images of odd aspect ratios fill the container. Nasty bit of code and it might be to blame for that.  
It's still there, but once all the thumbnails have been processed (which will be pretty soon, >500 to go!) I'm very much looking forward to taking it out and shooting it!

6 minutes ago, swjr-swis said:

Do you have a priority list of mods you want checked first?

No, that's the first thing I need!  Not been playing much modded KSP recently so bit out of the loop on which mods alter craft but don't add parts.  I know there one that add fuel to wings, but I've forgotten what it's called! 
But don't worry about it this week, tbh I got lots on too, so I'm unlikely to get round to that this week.

[Val]

2 hours ago, katateochi said:

I could do it two ways; the tool I'm using allows me to store several versions of the same image (ie at different sizes), so do I store two versions at different res (which more than doubles the storage overhead) or do I just store the high res (which means higher overhead in data transfer, but not double).  I can't decide which to do currently.

What costs the most? Bandwidth or Storage? I think you should go for multiple sizes.

Also small bug. Numbered lists get cropped. It's worst in Microsoft browsers, and not quite as bad in Chrome. Haven't tried other browsers.

Microsoft	Chrome

 

Edited February 16, 2016 by Val
Image fubar

[sal_vager]

Can you store one full sized image and resize it on demand?

Your storage should be equal to the single images + temporary cached images, though those will likely be in memory, but it would mean a delay before the end user received all images.

[swjr-swis]

46 minutes ago, Val said:

Also small bug. Numbered lists get cropped. It's worst in Microsoft browsers, and not quite as bad in Chrome. Haven't tried other browsers.

I'm seeing the same thing as the Chrome screenshot in Firefox, seems 2-3 pixels are axed on the left side.

[Val]

6 hours ago, katateochi said:

I may also have goofed a bit on the thumbnails.  I resized them to fit the size they are need on the main results page.  But on pages like hangars they are displayed larger so the thumbnail looks a little low quality.

They also don't fit the thumbnail frames so well. The last wide one when there's odd number of crafts.

[katateochi]

4 hours ago, Val said:

What costs the most? Bandwidth or Storage? I think you should go for multiple sizes.

Also small bug. Numbered lists get cropped. It's worst in Microsoft browsers, and not quite as bad in Chrome. Haven't tried other browsers.

Ah well spotted with the <ol> numbering getting squashed, guess I only check it with less that 10 items!

I thought that storage was going to be more costly but the sheer number of GET requests means that is the more costly.  I think I should store multiple sizes too (which paperclip, the ruby library I'm using for this, is very well setup for)

4 hours ago, sal_vager said:

Can you store one full sized image and resize it on demand?

Your storage should be equal to the single images + temporary cached images, though those will likely be in memory, but it would mean a delay before the end user received all images.

I think that the way the hosting is setup that would incur more costs because the full sized image would be the thing being transmitted on each request.  resizing on demand is sort of what I've was doing while the thumbs were on imgur and other sites, but with full or half res images that's still quite a lot being transferred out of the amazon storage.  The image store doesn't provide any caching and once an image has been pushed to the store, fetching it doesn't touch the KX server at all.  So caching has to be handled by a CDN (which I have setup, but right now I can't actually tell if it's doing it's job, going to submit a support request on that later).  The advantages is v low impact on the KX server, the disadvantage is it's all kinda confusing as I've not set something like this up before! 

I really should have thought to store higher res versions too and if I'd spent a bit more time getting to grips with this setup I'd have realised that thumbs for hangars and those narrow full width ones didn't look right.  But it was kinda rushed in order to get imgur to lift the block (which btw I now should be able to do as all, well all except 3 pesky craft, no longer have imgur hosted thumbs).

What I'm going to do is change it so that I store multiple sizes and fetch the appropriate size as needed.  For the most part that will be the low-res thumb, as they are going to be fetched more than anything it makes sense for them to be no bigger than needed.  I'll then store two other sizes to fit the other required sizes and maybe store the original full res too? If that's just being stored then that prob won't add much more overhead. The slightly annoying thing is going to be in setting up the urls for the other sizes without changing the existing ones, but I think I can work around that.

Thanks for your thoughts on this guys!

 

[katateochi]

@Val, @swjr-swis fixed that issue with the lists. 

I've also setup a patreon page (link in my sig) so I've adding links to that on the site too.  Also updated links from KerbalStuff to the new Spacedock.  

Still working on better processing of the thumbnails, but that's being a little tricky, so I thought I better get a few quick updates sorted first.