Basically what I meant Steve. I meant the people who devised and operated the system. They knew all codes could be broken and that Enigma had certain weaknesses. I doubt they were aware of how much their own procedures were broken or not followed properly allowing a "way in" to breaking the codes.
You are quite right. The problem was that they concentrated on the level of encryption, which would certainly resist any kind of brute force attack in their assessments.
They knew there were weaknesses. For example, throughout the 1930s the Germans continually made what we would now call hardware upgrades as well as changes in procedure which successively defeated the Polish methods of breaking Enigma.
On 15th September 1938, the Germans entirely changed the procedure for enciphering the message keys, and the catalogue method devised from 1936, became completely useless. It was this change that led to the development of the Zygalski Sheets and, most famously, the Polish 'Bomba'.
The Polish Bomba was based on the principle that the random 3-letter message key was sent twice at the beginning of each message and that every now and then, a particular plaintext letter, yields the same ciphertext letter three positions further on.
As an example, assume the indicator is A
WB T
WY. We see that the letter
W occurs twice in the second position. The codebreakers at Bletchley Park would later call such occurences
females. As both letter groups (AWB and TWY) originate from the same plaintext (e.g. ZXS), we know that the letter
X is encrypted into
W twice with an interval of 3 steps. This is a unique property that can only occur with a limited number of settings.
If enough
females were found, the Bomba could be used to recover the current Enigma settings in less than two hours. This is simply because not all possible settings have to be tested (which might take a very, very long time).
The Germans dropped this double encryption of the message key at the beginning of each message in May 1940 and rendered the Polish method useless once again.
The British 'Bombe' though based on the Polish system, used a completely different approach. It was based on the assumption that a known (or guessed) plaintext, a so-called
crib, was present at a certain position in the message. This is where signals intelligence was involved.
Though the Germans did upgrade their machines throughout the war, the KM added a fourth rotor for example, there were always problems introducing a new piece of hardware across the entire network. The four rotor M4 machines had to be backwards compatible with earlier three rotor machines. The rewireable reflector ( UKW-D) could have posed a fatal threat to Bletchley Park, had it been more widely introduced, as could the programmable cipher wheel (Luckenfullerwalze) which featured 26 user configurable notches and allowed the number and position of the notches of each wheel to be changed frequently. It was planned to use this in conjunction with the UKW-D. the Enigma-Uhr was introduced by the Luftwaffe in 1944 and should have caused some serious problems for Allied code breakers. I can't explain how it works in a reasonably short post, but it was initially compromised by poor operational procedure on the part of operators who didn't understand how the system was supposed to be used.
The Germans were not overconfident about the encryption of Enigma, they were overconfident in the ability of the hundreds or thousands of operators to use it properly and securely.
I hope I've attached an interesting Bletchley Park file about the Enigma-uhr, which will save a lot of typing
Cheers
Steve