Can I get some Linux Help and Advice?

[NewtSoup]

Hi there,

I'm asking here because basically the KSP crowd are generally nicer than the people I meet on the official Ubuntu forums, to the point where I deleted my Ubuntu One account.  I realise there may be some overlap of course.

I currently run Ubuntu 16.04 with the Gnome 3 Desktop Environment.

Ubuntu 18.04 came out six months ago and the software updater is finally suggesting I update ( decided to wait rather than trigger a manual upgrade )

However when I try it tells me there is not enough free space on boot.  Except there is.  I shall furnish people with a screenshot of the pertinent information

 

The installer wants 232 MB free space.  I have 244 MB free space.  apt-get autoremove won't do anything because I've already cleared space.

I created a live CD but the live CD says I have no operating system installed on SDA which I clearly have and it wants to do a new partition table and a fresh install rather than an upgrade.

I thought OK I shall just resize the boot partition.  When I installed 16.04 I chose the LVM option as it said "this will make it easier to resize partitions later".  It really doesn't all the options are greyed out.  Would be easier with standard Ext4 partitions!  Even when I deactivate the root partition / file system.

So, my questions are:

1) Why is it telling me I don't have enough space to do the upgrade?
2) How do I get around this issue?
3) Do I even want to upgrade to 18.04 ? I mean everything works great at the moment.

Your collective wisdom will be appreciated.

Apologies for not continuing my exploits in going everywhere, I've been rather ill and spent time in hospital.

[LordFerret]

I'm going to venture to say that the free space issue might be that Linux could be looking at free (empty) blocks, or something along those lines, and you're too close to meet that requirement. Just a guess mind you.

As for upgrades, if they're in Stable then yes. If not, then you can safely wait a bit.

How to get around... I've seen a few suggestions online... like moving usr & var elsewhere. Maybe this will help?...
https://unix.stackexchange.com/questions/92160/need-more-disk-space-for-root

 

[Harry Rhodan]

I would say you could free up some space by deleting older kernels. If you try to completely reinstall there should always be the option to manually select how to partition your drives or how to use existing partitions.

LVM would make it easier to resize stuff because you can freely move your partitions around on physical extents or even across physical extents. But you would still need room to actually manoeuver around. Either by resizing your other partition or by not using up all the space to begin with. And then you'd probably have to interact with lvm yourself because I don't know if gparted can actually handle it.

And for future reference I would advice you to not just make a /boot partition but to make a partition  for / and one for /home. That way you can quite easily reinstall the whole system without wrecking most of your data. A seperate /boot is (or was) only useful if you wanted to actually dual boot another OS from the same HDD. Wait, you need the /boot for the lvm.

Edited September 10, 2018 by Harry Rhodan

[Green Baron]

I think the others are right. It may be that e.g. for extraction of something not enough continuous space is available.

May i suggest for the future to do something like:

5-10GB swap partition, mount point swap

50GB root partition, mount point /

"many" GB home partition, mount point /home

moar by gusto :-)

And format / and /home with a file system like ext4 or reiserfs, available during partition creation. Ext2 is not up to date and has no journal, suboptimal in case of power out or similar ...

 

I use this setup since many years, it leaves the user data partition free of any system stuff, allows for a change of os, desktop, etc. without touching user data and up to now i never had a space problem (on the root partition, i mean :-)).

I know this is of little help right now. If you have a free harddisk somewhere you could plug it in, do the above or similar setup, give the user the same name and id as the one you already have, boot from the new one, mount the old one and then copy only your home tree. Copy as user, not as root.

No system upgrade is needed.

If you need any more help, don't hesitate and so on :-)

Edit: i assume you are not running Linux from an SD or USB drive but from a real harddrive. Else i am not sure if ext4 is supported ... maybe one of the cracks jumps in soon :-)

Edit: oh, what do you use lvm for ? If raid or mirroring, i'd just mirror the relevant data, that is the home partitoin. Another reason to split up :-) I did raid once but came away from it since it made more work than it had use. I had to read all the documentation again when i bought a new pc ... anyway, with Grub2 a boot partition for lvm isn't needed.

Edited September 10, 2018 by Green Baron

[LordFerret]

1 hour ago, Green Baron said:

i am not sure if ext4 is supported

USB drives can be formatted ext4, but you need to be wary of the fact that USB drives aren't often of the capacity they claim... which can cause unexpected issues/results.

[Nuke]

yesterday i stared at my raspi tablet terminal all night (i sshed in so i wouldnt have to use the chicklet keyboard and do a lot of squinting). so after updating the distro, i find out that there is a newer one out. after backing up the card i attempted to do the upgrade to the new distro in situ and it resulted in nothing working. i had doom, descent, an snes emulator, quake 3, etc all working perfectly fine, figured the upgrade process ruined it. so i re-flashed the card with a clean install of the new system. went through all the documentation (using linux is like reading 'war and peace' all the way through every weekend), recompiled all the things. and nothing worked. aparently they borked a lot of the gpu drivers in the new distro. so this morning i re-installed the backup on the old distro. now everything works again.

i also recompiled my i2c joystick driver, and that seemed to work right the first time. now i need to build a better one. its going to have a thumbstick, throttle slider, trackball, hall effect encoder knob, and 4 gateron (cherry knockoffs) brown keyswitches for the shoulder buttons and anything else i can cram in there, all powered by an atmega328p. now im starting to question whether or not i2c is fast enough to move all that data. one way to find out is to build something. its less frustrating than working with linux.

[stibbons]

On 9/9/2018 at 4:50 PM, NewtSoup said:

When I installed 16.04 I chose the LVM option as it said "this will make it easier to resize partitions later".

In general it's a good option, yes. But right now your system is set up with /boot outside the LVM. Once upon a time the bootloader was unable to read kernels from LVM volumes, so a separate partition to boot from was required. That's no longer the case, but I don't know if the Ubuntu installer still defaults to that layout or not. At any rate, resizing that partition isn't really an option for you unfortunately.

On 9/9/2018 at 4:50 PM, NewtSoup said:

1) Why is it telling me I don't have enough space to do the upgrade?

The two main causes of this I've seen were reserved space on the filesystem, and open file handles for deleted files.

The EXT filesystem allows to reserve disk space that can only be written to by the superuser (root). From the mke2fs man page:

Quote

       -m reserved-blocks-percentage
              Specify the percentage of the filesystem blocks reserved for the
              super-user.   This  avoids  fragmentation, and allows root-owned
              daemons, such as syslogd(8), to continue to  function  correctly
              after non-privileged processes are prevented from writing to the
              filesystem.  The default percentage is 5%.

The numbers in your screen shot don't seem to line up to account for this. But just in case, it may be worth checking the reserved block count by using the `dumpe2fs` command, which will dump information about the filesystem including the reserved blocks percentage, and the `tune2fs` command to (slightly) decrease the percentage on the filesystem. You're only a little short, so if it's set to 5%, changing it to 3 or 4 should see you clear for now.

Secondly, if one process has a file open and another process deletes it, the file will still be taking up space on the filesystem even though it can't be seen via ls or with disk space reporting tools. You can use tools like `lsof` to look for open files on the /boot filesystem, or if your machine has been running for a long time it may be easier to just reboot.

Seeing as the upgrade tool is reporting a 1KB difference, it may also just be because of block size issues.

2) How do I get around this issue?

Easiest way? Look at what's in /boot and clean it up. 242MB seems like a lot. I don't have an Ubuntu machine handy, but my Debian home server has the currently running kernel and a backup of the previous version, and right now it's taking up 43MB. Use apt to see what kernel packages (if Ubuntu is still similar to Debian, you can search for `linux-image-*` packages) are installed, and remove old ones. Even just removing one should get you under the limit.

 

Edited September 11, 2018 by stibbons

[NewtSoup]

Thanks for the ideas.  Sorry for being slow.  I forgot to hit the "Notify me of replies" button.

Yes I'm on a 500gb SSD not an SD card or a USB key. 
I've already removed all the kernels and headers except the ones I'm using.  (Or at least I think I have )
Moving USR and VAR won't help - they're inside the Logical Volume which is the one I need to resize, boot is a "regular"partition and is 500mb. 
I think a 5 - 10 GB boot partition is a tad excessive but 1gb is fair.

From what I can tell I need to reduce the file system within the LV first.  THEN reduce the size of the LV and then I can increase the size of /boot...  hopefully

 

[LordFerret]

I think you're going to want to boot from a live disc and run GParted from it. I think the changes to be made require the volume to be unmounted. Don't quote me though, it's been a while.

[Green Baron]

Yep, if the partition to resize contains the whole tree it must be done from elsewhere.

If all fails, i'd keep in mind that lvm is not needed for a simple resize (only if a partition spreads over several volumes). The files ystem tools can resize as well. Lvm is the right tool for spreading partitions over volumes, to setup software raid and for mirroring.

Edited September 13, 2018 by Green Baron

[NewtSoup]

Just before I do go messing around with a live disk - given:
 

ruth@ruth-ubuntu:~$ sudo apt-get autoremove
[sudo] password for ruth: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

AND

 

 uname -a
Linux ruth-ubuntu 4.15.0-34-generic #37~16.04.1-Ubuntu SMP Tue Aug 28 10:44:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

 

What can I safely delete from /boot?

 

 

Those package files seems to be between 10 and 60mb each.

 

 

Edited September 18, 2018 by NewtSoup

[Green Baron]

 

In principle you need the files that correspond to the version shown by uname as well as the memtest files and the grub directory. I would assume that besides kernel, ramdisk and config you want to keep the abi xxx and retpoline xxx files, but they seem to be ubuntu or setup specific ...

You need all the files that have 4.15.0-34 in them, plus memtest* plus grub. Maybe other versions if your boot menu shows them and you need them for fallback reasons or whatever ...

As an ubuntu user you can use synaptic i think. Search "linux-image-xxx" where xxx equals the version you want to keep delete ! delete !(*) and purge them. Once and for all.

And try to do different partitioning ;-)

Edit: i am amazed that apt autoremove doesn't clean that stuff up. Is it marked as being in use ? You could check that with synaptic as well ... in which case it might be wise to not just do away with it ...

(*) censored

Edited September 18, 2018 by Green Baron

[LordFerret]

1 hour ago, Green Baron said:

 

In principle you need the files that correspond to the version shown by uname as well as the memtest files and the grub directory. I would assume that besides kernel, ramdisk and config you want to keep the abi xxx and retpoline xxx files, but they seem to be ubuntu or setup specific ...

You need all the files that have 4.15.0-34 in them, plus memtest* plus grub. Maybe other versions if your boot menu shows them and you need them for fallback reasons or whatever ...

As an ubuntu user you can use synaptic i think. Search "linux-image-xxx" where xxx equals the version you want to keep delete ! delete !(*) and purge them. Once and for all.

And try to do different partitioning ;-)

Edit: i am amazed that apt autoremove doesn't clean that stuff up. Is it marked as being in use ? You could check that with synaptic as well ... in which case it might be wise to not just do away with it ...

(*) censored

Yes, synaptic can be used. If you run into problems installing it (which you shouldn't), there is plenty of help online (latest release of synaptic and ubuntu). Synaptic will clearly define any dependency issues, and show exactly what packages you've got installed... making uninstalls/deletion a breeze.

Thinking about it, I'm pretty sure you've already got synaptic installed, and gdebi as well. Two things you should look to become familiar with.

[NewtSoup]

That's a big help.. I only need 4mb of extra space to do a distribution upgrade.  I'll try synaptic and remove the earlier packages.  I was pretty sure I could remove the images that were not *.0-34 but I wanted to be sure before I did.

If synaptic comes up blank then I have copied the files I will remove manually to a live USB.  that way I can restore them if needed.

First though I have infinity war to watch with my son.  We've been lent the DVD and I've not seen it yet.  Also I've had nearly a whole bottle of Malbec.  Not a great time to go mucking around with /boot

[Harry Rhodan]

There's also a small tool to remove old kernels in the byobu package: http://ubuntuhandbook.org/index.php/2016/05/remove-old-kernels-ubuntu-16-04/

[NewtSoup]

On 9/13/2018 at 3:32 AM, LordFerret said:

I think you're going to want to boot from a live disc and run GParted from it. I think the changes to be made require the volume to be unmounted. Don't quote me though, it's been a while.

I did try that from a live USB but even then I could not work out how to get it to let me resize the logical partition.  All the options were greyed out. I can possibly do it with command line as I know the commands now.  First though. Synaptic.

 

Synaptic Says I have 4.15.0.33 and 34 installed.. but not any earlier ones.. yet I have files for them.

I am going to do what I said above.  Just backup one of the kernel images to a USB live stick, delete it from my HD and then reboot.

[NewtSoup]

Here we go I am running on 18.04 and I have to say it's an unqualified ....

DISASTER lol.  I have severe keyboard lag.  The screen flickers constantly.  Nothing works quite correctly.  I'm getting lots of " xxx has stopped working" it refuses to see my primary monitor.

It is just about usable so I am currently backing up various save games and documents and downloads and pictures and then I'm going to format and reinstall.  NOT using the LVM this time as I'm only ever going to be using  single disk

 

My advice on upgrading from 16.04 to 18.04 is don't  If your system works and you don't want to do  a wipe / install  then stick with 16.04 if everything's working fine.  Don't use the upgrade facility.

It's possibly all borked because I had removed unity installed Gnome 3 already in 16.04 as I didn't like the Unity desktop

[Green Baron]

I do not know your installation but it seems to me that something has been messed up long ago, which could be the reason for your packaging tool being out of sync with your installation. Maybe it was the desktop switch though that should work imo. At least it does in debian, but i read ubuntu is somewhat special with its desktop integratoin. Too much integration :-)

 

I think you are on the right track now:

Do a new installation. Forget lvm if you don't need it for spreading partitions over physical or logical volumes or use raid levels >2. Ext4 can resize as well, though i would just plug in another harddisk and file system links for shifting directories to other partitions if space gets narrow. And use rsync for mirroring.

Suggested partitions:

swap                                       5GB

/            ext4 formatted         50GB

/home  ext4 formatted         big enough :-)

 

If you have server stuff that you want to separate that'll be on an own partition or better, on an own machine.

This i run on my workstation since many years without any problems. If i mess up something (which i did in the beginning) i can at any time delete the system partition /, format and start anew without touching my data, savegames, photography stuff, programming nonsense, etc on the /home partition.

I know this is a workaround, not a solution :-)

[NewtSoup]

Hey, thanks for responding.  In the time since my last post I backed up all my important stuff - KSP directory, minecraft and road redemption and my docs and videos.. oops. I forgot videos.  ah well.. never mind.

Anyhoo I am now on a standard install of 18.04 and everything is fine again.  The videos directory contained some KSP stuff I was going to upload so I've lost a couple of recorded missions.  Never mind.

[LordFerret]

2 hours ago, NewtSoup said:

DISASTER lol.  I have severe keyboard lag.  The screen flickers constantly.  Nothing works quite correctly.  I'm getting lots of " xxx has stopped working" it refuses to see my primary monitor.

This smells to me of outdated or incorrect drivers. Linux is usually all-inclusive, to an extent, but sometimes you need to give it direction to the proper (older) driver. I've had to do this in the past, it's kind of required for older machines.

If you decide to try the upgrade again, first make note of the driver being used in this current working version - and then check again after the upgrade. See if the same driver is used.
Use the "lshw" command to show this stuff.

[NewtSoup]

3 minutes ago, LordFerret said:

This smells to me of outdated or incorrect drivers. Linux is usually all-inclusive, to an extent, but sometimes you need to give it direction to the proper (older) driver. I've had to do this in the past, it's kind of required for older machines.

If you decide to try the upgrade again, first make note of the driver being used in this current working version - and then check again after the upgrade. See if the same driver is used.
Use the "lshw" command to show this stuff.

If I remember I will.  Not planning on another upgrade or distro change for at least 2 years now.  Though I am going to stick the vanilla gnome desktop environment back on it.  I really don't like the Ubuntu look - yes, I could have gone with Mint or Debian, but I didn't.  Ubuntu with Gnome 3 is what I like

 

[NewtSoup]

Annoyingly on a brand new installation of 18.04 I'm getting random freezes.. Maybe after 5 minutes.. maybe after 15 hours but last night i had to hit the reset button on my PC 4 times.  I've not had to touch that button for years.

Doesn't matter what I'm doing - although If I'm doing nothing it doesn't happen.  I can leave netflix playing all night and it's fine. It seems to be when there's interaction between me and the PC.  The system will Freeze, I can wave the mouse pointer around, the keyboard becomes unresponsive and I am unable to interact with any windows with the mouse.  Any animated effects stop animating.  All I can do is move the mouse pointer and my only recourse at this point is to hit the physical reset button.

[LordFerret]

Normally I would be thinking dust or static discharges, but in your case maybe not. Still, check it out... is the fan(s) running? Do you see a dust issue?

 

Have you looked at any of the logs yet? Open a root terminal...

Start with:
'cat /var/log/messages'
(you might need to use  'cat /var/log/syslog' because Ubuntu/Debian)

If whatever caused the freeze generated a trappable error, it should show as the last line there.

 

You can also look at the boot log, but your system successfully booted (look anyway):
'cat /var/log/boot.log'

You can also look at kernel-related potentials here:
'cat /var/log/kern.log'

and
'cat /var/log/dmesg'

 

Does anything obvious stand out in the logs? Error messages or 'warnings'?

[Green Baron]

Jumping out of the window with:

"Video driver !"

(splatters on pavement)

:-)

Edited September 21, 2018 by Green Baron

[NewtSoup]

Hardware maintenance I know exactly what I'm doing,  right down to stripping and refurbishing fans.  Everything on that score is very tidy, cable routed for best air flow, large quiet fans all spinning as they should, no overheating at all.  

Right now it's been 20 minutes since the last freeze so digging through the logs is a chore I'd rather not do right now.  That said the logs app in Ubuntu has a section marked "Important" and it's contents is as follows:

15:51:40 kernel: nouveau 0000:01:00.0: msvld: init failed, -19
15:51:40 kernel: nouveau 0000:01:00.0: msvld: init failed, -19
15:51:40 kernel: nouveau 0000:01:00.0: msvld: unable to load firmware data
15:46:30 pulseaudio: [pulseaudio] bluez5-util.c: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
15:46:06 spice-vdagent: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0
15:44:38 hpfax: [1332]: error: Failed to create /var/spool/cups/tmp/.hplip
15:44:38 colord-sane: io/hpmud/musb.c 2101: Invalid usb_open: Permission denied
15:44:27 hp-firmware: hp-firmware[621]: error: Firmware download failed.
15:43:29 spice-vdagent: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0
15:43:28 colord-sane: io/hpmud/musb.c 2101: Invalid usb_open: Permission denied
15:43:24 hp-config_usb_p: hp-config_usb_printer[401]: error: HP Device plugin's are not installed. Please install plugin's using hp-plugin command.
15:43:23 kernel: ACPI Error: Method parse/execution failed \_SB.PCI0.SAT0.SPT2._GTF, AE_NOT_FOUND (20170831/psparse-550)
15:43:23 kernel: MODSIGN: Couldn't get UEFI db list
15:43:23 kernel: Couldn't get size: 0x800000000000000e

The top 3 at 15:51:40 are the ones that probably happened right before the crash.

The HP stuff relate to my printer - an ageing laserjet - those don't worry me and only affect printer functionality.

I don't think it would be my video driver - I'm using the Cannonical recommended and tested one from nVidia.

My system though a bit old is pretty solid:

Intel Core i5 3570k  - modest overclock to 4ghz has run reliably for over 3 years on this.
Gigabyte Z77 UDH3 Main board
16GB motherboard compatible ram - can't remember exact brand and I'm not pulling a stick to check.
nVidia GTX 660 Ti 2GB
Samsung 500gb SSD
Antex 600W Silent PSU