12

I have two Windows Server 2019 hosts that cannot start the OpenSSH SSSH Server service after the following Windows updates were installed:

.NET Framework 4.8
October 2024 Cumulative Update
Malicious Software Removal Tool

The Date Modified changed to last night on all files in C:\Windows\System32\OpenSSH:

OpenSSH Directory

Attempting to start the service results in:

"Windows could not start the OpenSSH SSH Service on Local Computer. Error 1067: The process terminated unexpectedly"

There is nothing in the SSH log file when attempting to start. The Event Log gives not helpful details that I can find other than saying "The OpenSSH SSH Server service terminated unexpectedly. It has done this 8 time(s)."

Greg Askew
  • 39,132
Sam Daniels
  • 123

9 Answers9

21

The permissions on the LOG directory was wrong. I removed the LOG folder, restarted the service and everything started working again. The LOG folder was automatically re-created.

But.. if you enter the logfile (as admin user), and after accepting "you need admin permissions to access this folder", your username end up in the list op users with read permissions. after that, the deamon won't start again. removing the user from the security list of the LOG folder solves this, and the service starts again. :(

From Microsoft December 2024 monthly update, known issues section:

https://support.microsoft.com/kb/5048661

Following the installation of the October 2024 security update, some customers report that the OpenSSH (Open Secure Shell) service fails to start, preventing SSH connections. The service fails with no detailed logging, and manual intervention is required to run the sshd.exe process.

Workaround:

Customers can temporarily resolve the issue by updating permissions (ACLs) on the affected directories. Follow these steps:

  1. Open PowerShell as an Administrator.

  2. Update the permissions for C:\ProgramData\ssh and C:\ProgramData\ssh\logs to allow full control for SYSTEM and the Administrators group, while allowing read access for Authenticated Users. You can restrict read access to specific users or groups by modifying the permissions string if needed.

    Use the following commands to update the permissions:

$directoryPath = "C:\ProgramData\ssh"  
$acl = Get-Acl -Path $directoryPath  
$sddlString = "O:BAD:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICI;0x1200a9;;;AU)"   
$securityDescriptor = New-Object System.Security.AccessControl.RawSecurityDescriptor $sddlString $acl.SetSecurityDescriptorSddlForm($securityDescriptor.GetSddlForm("All"))  
Set-Acl -Path $directoryPath -AclObject $acl
  1. Repeat the above steps for C:\ProgramData\ssh\logs.

Win32 OpenSSH Issue #2282 Opened October 8, indicating the issue began with version 9.4:

v9.4.0.0p1 and later enforce permissions on the logs folder, leading to undiagnosable crashes of the service after Windows Update #2282

https://github.com/PowerShell/Win32-OpenSSH/issues/2282

Linked to this pull request:

add check for prog data folder permissions during sshd service startup #686
https://github.com/PowerShell/openssh-portable/pull/686

Greg Askew
  • 39,132
Wig-B
  • 226
4

I encountered this on Server 2022. With help from this StackOverflow answer and Gustavo's comment there, to fix this:

  1. Right-click on the C:\ProgramData\ssh\logs folder and select Properties.
  2. Click on the Security tab, Advanced button and remove all users except SYSTEM and Administrators. Alternatively, downgrade other users to read-only.

The OpenSSH service should start now.

What happened (how to break it again)

If you're logged in as an administrator and double-click on the "logs" folder, you'll be offered the option to "Click Continue to permanently get access to this folder": SSH logs permissions

If you click Continue, the user you are logged in as is granted Full Control permissions on the logs folder. Apparently the October 2024 update makes a change that will refuse to start the OpenSSH service if any "extra" users have write permissions to the logs folder. However as @Kivioja Antti's answer reports, downgrading that user to read-only (Read & execute, List folder contents, Read) also fixes the problem.

Mark Berry
  • 343
  • 4
  • 18
3

Acording to CVE-2024-43615, the KB5044277 also adresses te mentioned OpenSSH vulnerability.

Might be worth to have a look at it.

However, explicit information about the update in connection with OpenSSH is still rare. Maybe more information will follow but I am pretty sure that your problem has something to do with that.

Manu
  • 994
  • 6
  • 19
3

I tested this out, and logs directory permissions seems to be the problem as Wig-B found out. I can add groups with Read&execute, List folder contents and Read permissions to logs directory security and thighs seems to work.

Default permissions are SYSTEM (full control) Administrators (full control)

Kivioja Antti
  • 31
3

tl;dr: delete or rename %programData%/ssh (for example: C:\ProgramData\ssh)

When I try net start sshd, I get the error as below

The OpenSSH Server service is starting.
The OpenSSH Server service could not be started.
A system error has occurred.
System error 1067 has occurred.
The process terminated unexpectedly.

I tried many methods, including restarting the service,

checking read and write permissions, and even considered reinstalling.

However, I finally discovered that simply deleting (or renaming) the %programData%/ssh directory and running net start sshd again could solve the problem.

Carson
  • 141
1

I just solved this.

  1. Remove-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
  2. Rename C:\ProgramData\ssh to C:\ProgramData\\_ssh
  3. Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
  4. Start-Service sshd
  5. Stop-Service sshd
  6. Delete c:\ProgramData\ssh\sshd_config
  7. Copy C:\ProgramData\\_ssh\sshd_config to C:\ProgramData\ssh\sshd_config
  8. Start-Service sshd

Not sure if 1 and 3 are needed.

Nikita Kipriyanov
  • 15,624
Soliman
  • 113
1

Same issue, but on windows 10. To solve it:

  1. rename the C:\ProgramData\ssh\logs folder to logs2.
  2. restart service.....

That will create a new logs folder and it will work fine

Keonik
  • 11
0

Had the same issue.

But the log-directory alone was not enough. You also have to make sure only SYSTEM can read all of the private-keys in C:\ProgramData\ssh\

Also if you get an Access denied at the Add-WindowsCapability you may need to reboot the server.

NiceRath
  • 11
0

I had the same issue after updating my Windows server 2019 with December's update.

Unfortunately, Microsoft's set of PS commands solution in their KB article failed with the error:

At line:1 char:1
+ $securityDescriptor = New-Object System.Security.AccessControl.RawSec ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

They also have a format error in their KB, merging 2 commands in a single one - note this could result with an unexpected behavior.

Eventually, I got lucky by manually finding the faulty ssh directory permissions having the additional group AUTHENTICATED USERS - after deleting the group, the service started correctly. The logs directory actually had the correct permissions and was unaffected by the update (so deleting it also did not help to begin with).

If you have any additional permission groups you are not sure of, starting the sshd.exe using PS and administrator permissions will work as well without messing with any directory permissions.

Both solutions are workarounds same as Microsoft's solution until they would release an official OS update addressing this issue formally.

Pizza
  • 173