M. Morris Mano
Dealing with computer architecture as well as computer organization and design, this fully updated book provides the basic knowledge necessary to understand the hardware operation of digital computers. Written to aid electrical engineers, computer engineers, and computer scientists, the volume includes: KEY FEATURES: the computer architecture, organization, and design associated with computer hardware • the various digital components used in the organization and design of digital computers • detailed steps that a designer must go through in order to design an elementary basic computer • the organization and architecture of the central processing unit • the organization and architecture of input-output and memory • the concept of multiprocessing • two new chapters on pipeline and vector processing • two sections devoted completely to the reduced instruction set computer (RISC) • and sample worked-out problems to clarify topics.
I want to go backwards and learn more about how compilers, processors and memory operate on my programs. I am also interested in the physics on which all of this depends. Any good references or books would be appreciated...
Pick up any of the Hennessy & Patterson books. Computer Architecture - A Quantitative Approach or Computer Organization & Design - The Hardware Interface
Pick up a book on "Computer Organization" or "Computer Architecture" on Amazon. This is what we used when I was in college. It's not too thick, and will give you the basics, from the gate level all of the way up to how memory is organized and programs are written. If, after this, you want to look deeper into the physics, then you'll want to pick up a book on semiconductor physics. (But if I were you I'd just start by looking up "logic gate", "diode", and "transistor" on wikipedia!)
Feynman has a nice bit on the Physics of Computation:
which addresses the second part of your question.
My first suggestion was going to be Code which has been suggested already. A better, but harder, book on the subject of processors is Computer Organization & Design by Hennessey & Patterson. You might look for an older edition on Amazon or Half.com. They'll be a lot cheaper and have basically the same information.
These will both teach you the basics of how a processor works, assembly language, etc. This will help you understand how your program will be interpreted and thus, what sort of performance bottlenecks might exist based on your design.