"Any sufficiently advanced technology is indistinguishable from magic."
In the grim darkness of the 41st millennium, everyone thinks machines have souls.
Even worse, they very well might be right. Because that is just how weird everything is in the big blackness of the grimdark future.
This isn't really that absurd a belief, considering how we tend to anthropomorphise things like computers and cars - ascribing them personalities, habits and so on. The idea of Machine Spirits is what develops when you let this rampage unchecked by common sense among a technocratic & theocratic ruling class. However, what's important to know is that prior to humanity's downfall and the Dark Age of Technology, artificial intelligence was widespread in what was a typically utopian future. Given that the Adeptus Mechanicus have a hard-on for hoarding ancient technology, it's not unreasonable to assume that particularly old and/or complex machines do in fact think for themselves. This turns into nightmare fuel when you realize that Mars is honeycombed with factories, warehouses and other facilities in varying states of disuse - facilities that were the battlegrounds between the loyal and heretical halves of the Mechanicus during the Horus Heresy. Hideous self-aware weapons of death may still stalk the lower levels, insane from 10,000 years of solitude. For the rest of the Imperium, Machine Spirits are a reason to maintain machines and treat them with respect - from rituals of cleaning to appease a simple Lasgun to the complex rites performed to reawaken the demanding and temperamental machine spirits of Titans.
There is also one other problem that one tends to forget--there actually are things called Machine Spirits. After the rebellion of the Men of Iron during the Age of Strife, Artificial Intelligence was declared "Abominable Intelligence" by decree of the Emperor of Mankind. To get around this, human brains were recycled and placed into the more complex machines (such as Robots, and shipboard AIs) which would then process the more complex information and control systems under the direction of a human. Being recycled from dead humans, they're understandably pretty cantankerous. Plus there are several signs that some advanced equipment, such as Land Raiders, Titans, and various old spaceships, have actual inorganic A.I. (albeit with an extremely specific scope of intelligence) These machines remain in operation despite -and indeed, because of- the Mechanicus' inability to fully understand how they work.
It is also implied sometimes (depending on the source) that Machine Spirits may be more real than first thought. There are examples of weapons that have no power source still being able to fire themselves. There was also at least one instance of a bunch of gears that were not connected to anything but each other (i.e. no motor) actually moving on their own after being talked to by a Tech-priest (maybe micro-motors? The Mechanicus is actually advanced). Or a gun that actually felt that it failed its master. Considering the Dark Age of Technology's feats, it is possible that dark age humans had successfully found a way to bend the warp into machines to give the machines their own "soul". This may also have helped cause the Iron War. After all, what does Chaos do with unprotected, unaware souls? Heh, yeah. Anyway, most "leftover" Machine Spirits tend to be a bit crazy, though, and using human brains merged with tech is safer because at least you know how the brain works. Hah, sort of.
In truth, and after checking most fluff material both from the Imperium's codices and Black Library novels you can find there is plenty of examples of Machine Spirits actively working along with Techpriests, Space Marines, Titan crews, etc, and they will show enough independence of thought to get mad if an enemy hurts their operators or know when to sacrifice themselves if there is no other way to get the task done. One such famous tale is the Land Raider known as Rynn's Might that belonged to the Crimson Fists Chapter, which survived the missile that leveled their fortress-monastery. Despite not having any crew on board, the machine spirit fought a solo war against an attacking Ork Warband, killing its Warboss and many of his followers overnight, before finally being destroyed.]
Some authors imply that the existing machine spirits are actually the remaining "Men of Gold", the only true and relatively benevolent machine spirits. These machine spirits are not particularly fond of helping their human caretakers as they have little reason to despite being extremely powerful and having full knowledge of the object they inhabit. Tech-priests rarely interact with these passive aggressive mechanical individuals and have almost no memory of the event, even so these spirits often require a lot of convincing just to make the on/off switch work.
Another interpretation is the machine spirit is the Dark Age of Technology's equivalent to Siri. It is an AI, but not a sentient one. STC technology was designed to be as inherently easy to use as possible, which would include making everything a smart device that could understand verbal commands. Not true AI, the smart devices could hear commands and try to orient themselves to do what their operator wanted, even if the right button wasn't being pushed. Now, keep in mind that the current Imperium is working on copies of copies of replicas of crudely reprinted versions of the original STC instructions (duplicated imperfectly by rote). There are still elements of the smart device listening and wanting to follow their human's commands, but they have a garbled understanding of vocabulary (or in another scenario, said human is speaking the incorrect language due to several thousand years of divergence). The Tech priests in turn have developed their rituals to be heard and understood by the machine spirits, unknowingly speaking keywords the device can try to decipher. In short, a Techpriest could be speaking instructions to Siri descendants in just about everything they do when it comes to tech.
There exists another possibility. Given how incredibly advanced humanity was, there could be micro-circuitry which, due to its sheer complexity, developed in such a way that it provided a sort of pseudo-brain for many pieces of technology in a sort of subconscious manner with a vague, general awareness. Techpriests might have discovered this on their own and found ways to interface and communicate with these semi-entities.
A final, stranger theory is that "machine spirits" are actually the work of the Void Dragon. He, and some Necrons, can directly control machinery. Combine this with how the creators of much of the more complicated machinery in the Imperium actually worship the Dragon under the guise of the Omnissiah and one starts to wonder if the Dragon has something to do with all of this. Perhaps a mutually beneficial and permanent deal with the Emperor to protect human tech from being corrupted by Data Daemons in exchange for passively feeding on it?
Hell it could be a case of all of the above depending on the individual piece of technology. Which would go a long way to explaining why the Mechanicus have such a hard time. The Rules keep changing on them drastically and they have no idea there's more than one set of rules.