MSIX Remote Packaging for MSIX Bulk Conversions

I wasted a whole weekend debugging MSIX remote packaging for MSIX bulk conversions.  And I’m still pondering if it’s a bug or a feature!!

Weekends are for family; not for debugging “work” issues, writing blog posts and creating instructional videos.  But since I encountered an annoying issue late on Friday, I couldn’t help but take my laptop into the living room (make a mental note of what I just said) on Saturday morning so I could debug this niggling issue whilst spending time with the kids!

It was an encouraging start to the session when my daughter said “Dad, what are the only bees that produce milk?”.

“Finally” I thought “the kids are absorbing interesting facts from those David Attenborough documentaries…”.

Until she responded with “Boo-Bees (Boobies)” and ran off.

To be fair, it was a welcome distraction from sifting through Process Monitor logs.  Albeit not the response I was looking for.

MSIX Remote Packaging for MSIX Bulk Conversions

The issue I encountered on Friday was simply trying to initiate MSIX packaging on a remote, domain-joined machine in my Hyper-V lab.  It was prohibiting me from testing some bulk conversion logic on remote machines.

It was successfully launching an RDP desktop session to my MSIXWorker machine, but nothing (significant) seemed to be happening, and my MSIX Packager on the “controller” machine was stuck on “Preparing your virtual machine”.

I double-checked the instructions described here, whilst manually opening the inbound port 1599 for the MSIX remote service, but still no joy.

I decided to try using the command line by outputting a verbose log to see if I could glean more information. The typical command line for converting an MSIX package on the command line is (I will avoid discussing the XML template in this blog for the purpose of brevity):

msixpackagingtool.exe create-package --template "[path-to-xml-template]" -v

When I ran this command, everything looked to be working but then it just hung on “Activating IServerLifetime remotely”.  I assumed this is what the GUI version of the MSIX packager was doing too:

Now I’ve written service endpoints before (I wrote an App-V conversion service endpoint many years ago as part of Virtualisr), and this stunk of either the endpoint not starting or not being accessible.

In the verbose log it stated that it was trying to create a scheduled task that invokes “MsixRemoteServer.exe” but “failed” because /ST is earlier than the current time (spoiler – it didn’t fail – read on).  Hmmm.  Were my VMs out of sync?  Nope.  They were both setting the date and time automatically.  But surely if the MSIX remote server wasn’t starting, then the endpoint wouldn’t be listening to my remote calls and hence the process would stall?  It made sense.

I verified that MsixRemoteService.exe was copied over to my MSIXWorker machine.  I then ran a quick process monitor and could see that, whilst it was being copied over, it was never actually being invoked.  Again, it made sense.  But why wasn’t it being invoked?

(Fast forward many hours of Wiresharking, setting WinRM trusted hosts, telnetting to different ports, creating HTTPS listeners with self-signed certificates, standing up “Quick create” Hyper-V vanilla MSIX packaging virtual machines (that are apparently configured to work out of the box), using Test-WSMan to successfully verify HTTP, HTTPS and Kerberos PowerShell sessions, Anti-virus and Firewall exceptions and debugging group policies etc).

I needed a break.  So I took my girls to the golf driving range to take out my frustration on those tiny dimpled fairway pearls.  And it dawned on me that most of my debugging was leading me down the wrong path; Kerberos authentication doesn’t require trusted hosts configuring (domain-joined hosts are already trusted).  And I shouldn’t need to configure HTTPS listeners since my machines were both domain-joined and should authenticate remote sessions over Kerberos and not HTTP(S).  And this way of thinking led me to a eureka moment and a different angle of attack.

When the MSIX packaging process fails, it strips off most of what it creates on the remote machine making it difficult to debug.  So on Sunday morning (I ran out of time on Saturday), I decided to use Process Monitor to debug the scheduled task mentioned in the verbose log, by filtering on schtasks.exe.

I could see that (even though at this point the packaging process had failed and the task no longer existed) it was trying to create a scheduled task using the following command line:

"C:\Windows\system32\schtasks.exe" /Create /SC ONCE /ST 00:00 /TN d6107067-d055-4540-b23f-bb703fe7c594 /TR "C:\Users\alkaneadmin\AppData\Local\Temp\MsixRemoteServer\Bin\MsixRemoteServer.exe 1599 300 8eaedc94-5d3d-46fb-b8f9-dba8e7b402e9 DA905BDA83BA3593E6DFA1FE3FB5CE20A328C0A5 False" /RL HIGHEST /IT

When running this command manually (via a PS Session from my controller machine, to try and replicate the real-world scenario), I too received the warning about it failing since “/ST is earlier than the current time”.  But in reality, it didn’t fail.  It still created the task.  But because it was configured as a one-time only task, and the start time was in the past (00:00), it told us that the task had failed (a red herring). All the MSIX packager was trying to do here was to create a scheduled task with the aim of invoking it at some point in the future!

So I opened task scheduler on the worker machine and checked each tab of the task.  I correctly witnessed that the scheduled task could be run on demand, which was encouraging:

But what I witnessed next left me cold.  It can only be described as a feeling of momentary bereavement and loss.  Loss of time.  Loss of sanity.  Loss of faith (in Microsoft).  I felt sick, and my Robusta-infused latte tasted even more bitter.

I knew instantly that this was the culprit, because this issue caught me out many years ago when I created a scheduled task that didn’t work for “some” home workers VPN’d in:

This dastardly setting (set by default, I might add) will only invoke a scheduled task when the machine is plugged in to mains power.  So if you are sitting on your laptop using battery power, it will not invoke!!

I scuttled back into my office and left the kids binge-watching their one-thousand and fifty-ninth consecutive episode of The Loud House, and I plugged my laptop in!

Bingo!  Everything breathed into life!  The “Create package on a remote machine” function of the MSIX Packaging Tool started working, and so did command-line creation of MSIX packages!!

I had wasted a weekend of my life debugging an issue because my laptop wasn’t plugged in to an AC power outlet! (Hands over face, collapse to floor, roll over a few times and stand back up).

Is this a bug in MSIX packaging tool process?  Or a feature?  Does the “AC Power” box need to be unchecked so we can perform MSIX packaging on a remote machine whilst not being shackled to our office desk?

For anyone like me doing some development work using a “local” home lab, beware the (lack of) AC power!