SpaceDock.info (Mod Hosting Site)

[VITAS]

thx. ill see what i can find later.

I did divide the services between multiple servers in the past week so my guess is: i broke something in the process.

[SmashingKirby148]

58 minutes ago, RealGecko said:

It looks like this:

No additional info is provided.

 

Yup, that's what I'm getting too.

[VITAS]

@RealGecko So its a new mod? FlightPlan has 1.0? When did the error happen ( after uploading?)?

 

i have:

17K Nov 12 00:52 FlightPlan-1.0.0.zip
328K Nov 12 00:53 FlightPlan-1478908406.4358277.png
5.2K Nov 12 01:22 thumb_FlightPlan-1478908406.4358277.png

Ill check the db when i get home, but the file creaton dates are quite old (i dont know if the server uses your creaton date or does its own timestamp).

 

If someone else created a mod in the last 3 days or so please tell me if you got the same error or not.

Edited November 14, 2016 by VITAS

[RealGecko]

1 minute ago, VITAS said:

When did the error happen ( after uploading?)?

I created mod as usually, added header, updated info, everything was fine at this point. But as soon as hit publish I got Internal Server Error. That's true for both mods.

[VITAS]

ok so my guess is ive to see what the db says. (i assume the filesize is correct).

Btw found an unrelated bug: i get 401 unauthorized when i try to publish your mod as admin

 

Update:

i created my own test mod and it went without a problem.

Have you tried deleting and recreating your mods?

also please creat some differently named test mod and try to publish it.

 

If someone else created a mod in the last 3 days or so please tell me if you got the same error or not.

Edited November 14, 2016 by VITAS

[SmashingKirby148]

5 hours ago, VITAS said:

ok so my guess is ive to see what the db says. (i assume the filesize is correct).

Btw found an unrelated bug: i get 401 unauthorized when i try to publish your mod as admin

 

Update:

i created my own test mod and it went without a problem.

Have you tried deleting and recreating your mods?

also please creat some differently named test mod and try to publish it.

 

If someone else created a mod in the last 3 days or so please tell me if you got the same error or not.

 

I'm still getting it, even since yesterday. I'll try what you recommended to Gecko.

I have reuploaded it, getting the same issue still.

[RealGecko]

OK, as soon as "Add mod to CKAN" is checked you cannot publish or edit already published mod. Every time I push "Publish mod" or "Save changes" I get Internal Server Error. If "CKAN" is unchecked then everything goes smooth.

Problem is that there's no "CKAN" checkbox for saved, but not yet published mod.

[SmashingKirby148]

7 minutes ago, RealGecko said:

OK, as soon as "Add mod to CKAN" is checked you cannot publish or edit already published mod. Every time I push "Publish mod" or "Save changes" I get Internal Server Error. If "CKAN" is unchecked then everything goes smooth.

Problem is that there's no "CKAN" checkbox for saved, but not yet published mod.

 

Oh wow, you'e right. Just unchecked it and my mod went up perfectly. :S

[VITAS]

Found the problem (i think). will fix it as soon as possible. Ive to check some things first so i dont make it worse

 

Complicated explanation:

THe setup now is: Server with apache/db -> multiple servers with multiple gunicorn (app) instances

i missed to sync the netkan folder between machines so if you switch servers while publishing a mod it doesnt find the netkan file it generates for ckan and errors out.

Ive to sync all folders between each other before i can redirect them to one most up to date central share.

 

Edited November 14, 2016 by VITAS

[RealGecko]

1 minute ago, VITAS said:

Server with apache

Just curious: why Apache? Nginx holds high load better

[VITAS]

because i know it inside out.

[Thomas P.]

4 minutes ago, RealGecko said:

Just curious: why Apache? Nginx holds high load better

Because we support the preservation of american minorities

Jokes aside, VITAS knows it, and he runs the server, so why force him to use something he doesnt know.

(Not that I wouldn't fully support nginx for spacedock...)

[VITAS]

i dont feel forced. (how could you? "i dont visit apache served websites" ? )

[RealGecko]

2 minutes ago, Thomas P. said:

VITAS knows it, and he runs the server

Nice, it's good to have business with professionals

2 minutes ago, VITAS said:

"i dont visit apache served websites" ?

I worked with Apache and it showed itself memory hungry at high load. Switched my server to Nginx years ago

[VITAS]

Im old. no either i have menough memory to spare or gunicorn is a way larger problem. but i dont feel it. (but you still might be right)

[RealGecko]

Just now, VITAS said:

gunicorn is a way larger problem

Man, you intrigued me

Spacedock is open source, I think I'll try deploying it locally this week and play with it a little

[VITAS]

there are two versions: the current production version (based on kerbalstuff code and its a mess)

and the new version (two parts backend in python by thomas and frontend in slim3&jquery by me) thats still WiP

 

Im currently rsyncing the folders. So hopefully we will be ready to go for your mod in a couple of minutes

 

have to restart the site :/

 

Site is up again and bug fixed. ( i took the liberty to publish your flightplan mod)

 

also if you want to have some insigt in how i do things:

because gunicorn wont propperly use all the cores (and now its even distribute among several servers) i used apaches loadbalancing function with sticky routes and session cookies (gunicorn nromaly should stick with one server per usersession)

then i use reverse proxy to direct the app traffic to each server via openvpn (the mains erver and the app servers are in different locations)

i also use xfiles and other tricks to serve static content without gunicorns help directly from apache.

i also tried putting apache traffic server in front of it but it slowed down the site to much so im still runnign without caching server in front but i want to change that.

To figure all of this out took me a year. Now the cpu of the main server is down to 30% with about 250 ppl on the site. But loadingtimes arent not where i want them to be.

So the plan was to split front and backend to have the frontend (php,js,html...) run on cheap webspaces close to the users and do geoDNS (as weve done in the past with the modfiles). the backend will run in an similar setup as the site is now. Requests between the frontend and the backend will be done directly via ajax to the backend (the frotnends php framework serves only as routing and template engine)

Side effect: ppl could donate a gig or two of php webspace and we could have a mirror there. (sadly there are next to none python/flask hosters)

 

p.s.I proudly present to you a post youll never find on a minecraft forum

 

Edited November 14, 2016 by VITAS

[VITAS]

moved to upper post

Edited November 14, 2016 by VITAS

[legoclone09]

With regards to names, maybe have a back-end name for use in URLs and filesystems, that is the original name of the mod when it was first uploaded, but let you change the name showing on the website?

[VITAS]

could be done but im not messing with the current prod code unless it has showstopping bugs. i devote my time to the new code.

 

Btw @legoclone09 http://forum.kerbalspaceprogram.com/index.php?/topic/151880-for-all-streaming-folks

Edited November 14, 2016 by VITAS

[RealGecko]

Oh, sorry, guys. @allista released an update for Hangar mod, I was in nirvana for a moment.... for a very long moment....

1 hour ago, VITAS said:

Site is up again and bug fixed.

Thanks, works like a charm

App architecture is really tricky as far as I can see, but there are always ways for optimization. I'll write you PM, maybe we can discuss ways of making things better

[RealGecko]

By the way: NetKAN indexer always complains about license added from space dock. Spacedock says license GPLv3, NetKAN expects GPL-3.0. Now something is wrong with Overdrive's BSD License.

Any ways to fix?

[VITAS]

ill redirect that question to thomas and ckan.

If its on spacedocks end: youve the sources fix it :>

[Ruedii]

The on whole Apache vs. NGINX argument, it completely depends on the server load and server capability. 

Apache uses older, proven thread-based theory, and is designed to be used with a secondary caching content delivery proxy.  This works well for reliable medium-load servers.   However, this method tends to have a memory/bandwidth tradeoff.  This means to achieve more bandwidth, you need more and more memory.  This is not a problem until you are needing to serve while over 10K clients at once with dynamic assets.  Since most of the assets on SpaceDock are static and thus cache friendly, this shouldn't be an issue.  Under normal expected loads, Apache is more reliable and faster in many cases.  (This is why so many sites use it, or variants of it.)

NGINX works on an instance based system.  While this can handle heavier loads and more scalability, and is friendlier for high-end cloud-based and distributed servers, and interfaces better with modern CDN networks.  It's initial resource footprint is a little higher, but it's footprint increases a lot more slowly with additional transactions, because transaction data is kept a lot shorter, and most instances rely on shared memory for most of their data.  It is also a little more complicated to set up.  It's certainly a better option for many uses.  Spacedock might be able to benefit, maybe not.  It really depends on how the code is written, and what load it's under for what types of resources.

As a note, I strongly oppose running Apache without a caching proxy.   A caching proxy is usually a special "static data only" lightweight server.  It tends to have a much lower footprint than either NGINX or Apache, however it is very limited, being restricted to static assets, and sometimes basic non-database PHP processing.

[RealGecko]

Yep, both servers proved to be reliable each in it's own field, but IMO setting up Nginx is not harder than Apache, especially if we speak about serving static